:W&Bᾘk¤gM gzvulb17ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0¤%Õü‚" qm78tsw6ulws gzvulb17ŠØ gzvulb17juju11-uc-san-diegoSPAtop"› _wandb‚{}  batch_size‚256  dataset_limit‚0.2 dropout‚0.009968960775016434 epochs‚10  hidden_dim‚256 ' learning_rate‚0.0002045704266027611  num_attention_heads‚8  num_branch_encoder_layers‚5  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_vanilla_gzvulb17b88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8Š ôýäÔÐ×°Üš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64° 7 7 gzvulb17&  _runtime‚0  _wandb.runtime‚0t:“–jÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ôýäÔ Èë¶*-of*=/data/spatop/options_files/sweep_tttt_jj/config_gzvulb17.json*-l* /scratch/logs*-n*jjtttt_vanilla_gzvulb172-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8r/opt/conda/bin/pythonº 0z8n03p5nlj4ltfrvq2ywca0gcilt7s0¬˜Å¬ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€àòˆ¤ª€àãÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-78f24734-e3a1-59a7-ce1b-bb4283f77c97Ê13.2º 0z8n03p5nlj4ltfrvq2ywca0gcilt7s0µª½Ç2  requirements.txtmRçŒW gzvulb17ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64Y·åY gzvulb17ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_641›Æb*  gzvulb17b trainer/global_step2<ýÒ±\  gzvulb17ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64ºê/†/  gzvulb17b*"trainer/global_step2Éè5K  gzvulb17j: õýäÔÀö÷×*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] @é•’  gzvulb17j€  öýäÔ¨«É>î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 7.9 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 183 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ pjj[ gzvulb17jÉ öýäÔðŸÂ?·Trainable params: 194 M Non-trainable params: 86 Total params: 194 M Total estimated model params size (MB): 777.876 Modules in train mode: 5157 Modules in eval mode: 0 Total FLOPs: 0 BÌép gzvulb17j_ öýäÔ˜ýä?Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). f~ÌHz gzvulb17ji öýäÔð˜î?Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). 8ØŽËx gzvulb17jg öýäÔà‘€@Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). Òï O gzvulb17jn öýäÔØ̉@_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ¸9* y gzvulb17jh öýäÔ°è’@Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). ¼{&s gzvulb17jb öýäÔ¸ý˜@Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). H¢ˆwp gzvulb17j_ öýäÔ ðŸ@Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). :n½Þs gzvulb17jb öýäÔØø¤@Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ¦…7—m gzvulb17j\ öýäÔÀñ©@Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ¹ë5Ól gzvulb17j[ öýäԈˮ@Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ’ùb gzvulb17*Ð  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"MVTšê gzvulb17jØ öýäÔðû¸FÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. +ˆÙÆM gzvulb17j» öýäÔÀáÆF«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ïtŽûê gzvulb17jØ ùýäԨĖhÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. zcO gzvulb17j½ ùýäÔę̀h­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. [99Ü’‚(:ˆ „þäÔÿ¥—µ gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.80622863769531* gpu.0.memoryAllocatedBytes‚ 29039329280 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªdY; ‚(:– „þäÔ—Î…» disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚0% disk.nvme0c0n1.out‚75.15771484375$ memory_percent‚95.34611701965332 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚33537% disk./.usageGB‚265.02090072631836 disk.sda2.in‚0 disk.nvme0c0n1.in‚4.078125' proc.memory.percent‚18.023681640625 network.recv‚39262 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚0.421875 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0 disk.dm-0.out‚75.15771484375 disk.drbd2265.out‚0 disk.sda3.out‚0.421875' proc.memory.availableMB‚ 762.4921875 proc.memory.rssMB‚2953 cpu‚25.68673073333248* disk./.usagePercent‚17.785775015090106 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚4.078125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚0 disk.dm-1.out‚0.421875 disk.drbd2266.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚75.15771484375 䛄’ ‚(:ˆ “þäÔ£µ¢µ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.81355285644531* gpu.0.memoryAllocatedBytes‚ 29045620736 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.0754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:k}y/!‚(:¥ “þäԳĒ· disk.sda1.out‚0 disk.nvme0n1.in‚4.078125 cpu‚25.51917849326228 disk.rbd1.out‚0& disk.nvme0c0n1.out‚117.79052734375 disk.drbd2265.out‚0$ memory_percent‚95.34730911254883& proc.memory.availableMB‚ 762.296875! disk.dm-0.out‚117.79052734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 1.0078125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚4.078125 disk.drbd2265.in‚0 network.sent‚50520* disk./.usagePercent‚17.785642147179836 disk.sda3.in‚0 disk.sda3.out‚ 1.0078125 disk.sda.out‚ 1.0078125# proc.memory.rssMB‚ 2953.04296875$ disk./.usageGB‚265.0189208984375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚57888 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.078125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚117.79052734375) proc.memory.percent‚18.02394390106201éJÿÂ’"‚(:ˆ ¢þäÔÒÜ´ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.81843566894531* gpu.0.memoryAllocatedBytes‚ 29049815040 gpu.0.temp‚47.0 gpu.0.powerWatts‚297.097' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü!|,#‚(:¢ ¢þäÔ´ñÁ· disk.drbd1716.out‚0 disk.sda.out‚ 1.6015625 disk.dm-1.out‚ 1.6015625$ disk./.usageGB‚265.0320014953613 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0$ memory_percent‚95.34792900085449# disk.nvme0n1.out‚163.3017578125 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚4.078125 disk.dm-0.out‚163.3017578125' proc.memory.availableMB‚ 762.1953125 disk.sda2.in‚0 disk.rbd1.out‚0! proc.memory.rssMB‚ 2953.078125* proc.memory.percent‚18.024158477783203 disk.drbd2265.out‚0 disk.drbd2266.in‚0 cpu‚25.41940750026843* disk./.usagePercent‚17.786519997014747 disk.sda3.out‚ 1.6015625 disk.drbd2080.out‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125% disk.nvme0c0n1.out‚163.3017578125 network.sent‚63649 network.recv‚75952 disk.sda1.out‚03c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPªèƒŠ-&) _wandbruntime‚51  _runtime‚51w”ƒ¼'· _runtime‚ 51.184887638 _step‚0 $ lr-AdamW/pg1‚8.2911091013526e-06 $ lr-AdamW/pg2‚8.2911091013526e-06 trainer/global_step‚49 % _timestamp‚1.7884280728297646e+09î©C–¾(¹ $ lr-AdamW/pg2‚8.2911091013526e-06 trainer/global_step‚49 % _timestamp‚1.7884280728297646e+09 _runtime‚ 51.184887638 _step‚0 $ lr-AdamW/pg1‚8.2911091013526e-06(?ÞËl)bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP‹Z=rp*bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPÚHem+bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPò þ],bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP3Na¡d-b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPðè]xj.bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP Ól,e/ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP')‰‚g0bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPJæ%'n1bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP¸élj2bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPÐM–o3bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP”L9^4bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP´¡tÿs5bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP:YR¤n6bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP«æ™Éf7bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPæ8¹sj8bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÕ VMm9bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPú8ah:bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP"[!ßf;bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP“£á±hb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPÑ&Îâe?ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP VÃi@be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP‰;‰ÓbAb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPYÉ7hBbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP틤lCbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP¿†6»kDbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP@baEb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPävugFbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPSÈýnGbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP` ÅnHbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP<ÿ0¿vIbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP­ÛÝ«sJbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP1ÏÈ_Kb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP}’zhLbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP4‚Ÿ*rMbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP«r¡êkNbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP¦{:jObf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP_“e%qPbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPñ0‡iQbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP³!·qoRbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPÚݬúvSbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP‘ÏøkTbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPkt”®sUbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPV“=ãkVbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPªqk5lWbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÄÕó>tXbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP?d_hYbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPž ®ËoZbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP)a(©n[bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP©¼zj\bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP]î(e]ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPÁäô³c^b_ 4gcbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPbjšqdbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP;–Çlebh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP‚þ¢_sfbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP´ÆWigbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPŸÍxhbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP2,1œgibc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP}Ø€ïpjbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP)kî\okbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP¾ÑKÃflbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP…Jékmbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP='®inbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPk¾k&hobd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPÃë'npbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP¦‰õ›nqbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÒ!nrbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPX²lsbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP±Z«áptbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP<®Âbub^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPŹŸpvbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP¢Iâ^wbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPnÕ–nxbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPÿûÜÖjybf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPæHØezba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPjÀWf{bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPpÜ&dq|bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPY«ã q}bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP´Sƒx~bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPcn¦Ìibe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPw 8o€bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP³Psbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPþQÃsd‚b_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP»ûo…bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP¹é`šr†bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP3+êŸb‡b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP«z}=nˆbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÁâm‰bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP=¹œÂtŠbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPíê;s‹bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP¾ÇÈnŒbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPÜK&kbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP郼ƒjŽbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP¢‘jbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPÆ—E gbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP¬ggGq‘bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP>ȳˆn’bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPb¢7g“bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6719926765568 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚516687 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5828045605376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6616186559488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6510375590912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _runtime‚ 51.185816972 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1071987 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28502392832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28502392832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3930062848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3930062848 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471693 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1633 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79721472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5678995552768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1874 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1945304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1071987 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 103562229082 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26717 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 106844832768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685540 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚873311 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚873311 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚904 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6613912392026 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1874 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26464 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6723057783808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6617041281370 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28502392832 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877864448 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚384976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]LDeviceStatsMonitor.on_train_batch_start/requested_bÚ¼1Áytes.small_pool.allocated‚ 106665690458 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3176 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6719926765568 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚516690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1874 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6616186559488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877864448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 318474240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3192754176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 148970331136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6510350162944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1050379 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5679023687168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864Ô‰Àa´'¦®O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _runtime‚ 51.185816972 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1071987 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28502392832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28502392832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3930062848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3930062848 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471693 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1633 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79721472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5678995552768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1874 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1945304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1071987 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 103562229082 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26717 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 106844832768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685540 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚873311 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚873311 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚904 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6613912392026 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1874 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26464 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6723057783808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6617041281370 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28502392832 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877864448 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚384976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 106665690458 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3176 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6719926765568 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚516690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1874 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6616186559488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877864448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 318474240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3192754176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 148970331136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6510350162944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1050379 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5679023687168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚385480 _step‚1 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6719926765568 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚516687 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5828045605376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6616186559488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6510375590912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984¢$Áo§bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP¤ ]´s¨bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP„ê=»p©bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÙ*¬nªbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPªÈêÃu«bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPrøÊÆ]¬bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP2=0o­bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPM5Å`®b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPºt1Ãi¯bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP7¨Ñm°bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP¤5j±be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP:}$Aw²br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPÙÌgm³bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP==(Yj´be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPÔ\”Êfµba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP‘fÙ[o¶bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP!.éf·ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP„8Pyf¸ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP°YYe¹b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP‚êûø^ºbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP|'c»b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP…Oj¼be BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÌlPl½bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPpñch¾bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP·Mò e¿b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPÒú—gÀbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP!Ñu(nÁbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPŒßQrÂbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPqVükÃbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPf÷bžkÄbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP½–‡ølÅbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPëÄkÆbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPpeÿã^ÇbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPÇMHÚbÈb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP'—æeÎb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPó’¬mÏbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPøzÝÒgÐbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPÇ­šhÑbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPŸyr+rÒbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPA´©hgÓbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPv µyrÔbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP/ÆãbhÕbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPC;¿qbÖb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP¸Ça\×bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPƒ=AUVØbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPæœ ,oÙbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPšÖP‹gÚbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP¡YrÛbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP¥ÓBdÜb_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP>/Q¨sóbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPz«môbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPÝÎ jõbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP_!Ÿhöbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPÄSIb÷b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPº„~0løbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPOAU_ùbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP÷;*õdúb_ ¦mýbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¨j’hgþbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP>ë²iÿbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPjÖuÐi€bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPË^gçlbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP!ïôw‚br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPÓµò—hƒbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPÈs1i„bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP%šOîp…bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPÛ š$j†be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPÝ$Em‡bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP<^òvfˆba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP¯­Ónu‰bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPþùá˜cŠb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPþ!#a‹b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP}KrmŒbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPÝA9ékbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP¯¦…½eŽb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP˜šõº^bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP—³.1eb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP_±®d‘b_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP]u®l–bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPu—{_—bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPÄQí`˜b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPðyO"h™bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP… qšbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPOä¸pj›be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPK!†,jœbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPO 'õfba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPGžŠ^ožbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPòMS{lŸbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP\0Gce b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP4K‰Ôp¡bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPÜ$rJ.¢) _wandbruntime‚52  _runtime‚52­è«-£´M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6856664199168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6747707039232 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6745418186342 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 108774483766 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1983704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 318474240 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1983704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 151936521728 % _timestamp‚1.7884280737375507e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚890601 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1633 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3176 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1071469 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚919391 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚526943 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28502392832 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1874 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6747707039232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3930062848 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚890595 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 trainer/global_step‚49 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5791406416896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 105852519936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 52.093277152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 108957159936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚919886 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 151990192640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚492 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1874 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1962064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28502392832 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6853533167616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6747680647680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3192754176 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 105852519936 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5943342938624 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6748547075894 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3176 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1962064 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3176 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1093103 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0eùm5¾&¤¸M D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚890601 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1633 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3176 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1071469 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚919391 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚526943 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28502392832 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1874 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6747707039232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3930062848 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚890595 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 trainer/global_step‚49 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5791406416896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 105852519936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 52.093277152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 108957159936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚919886 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 151990192640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚492 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1874 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1962064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28502392832 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6853533167616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6747680647680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3192754176 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 105852519936 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5943342938624 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6748547075894 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3176 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1962064 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3176 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1093103 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6856664199168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6747707039232 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6745418186342 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 108774483766 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1983704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 318474240 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1983704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 151936521728 % _timestamp‚1.7884280737375507e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912G¯CvA¥b< loss/FBt3/assignment_loss"trainer/global_step2:JPjðuU@¦b; loss/FRt1/detection_loss"trainer/global_step2:JP*ÃóÕ@§b; loss/FBt3/detection_loss"trainer/global_step2:JPŸõ' -¨b( epoch"trainer/global_step2:JP• $ÂB©b= loss/SRqqt3/detection_loss"trainer/global_step2:JP6ðÌ6Aªb< loss/FRt1/assignment_loss"trainer/global_step2:JPèêjA«b< loss/FRt3/assignment_loss"trainer/global_step2:JP…³ûÁC¬b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP<—¹þA­b< loss/FBt4/assignment_loss"trainer/global_step2:JPÂH3B®b= loss/SRqqt1/detection_loss"trainer/global_step2:JP°£²@¯b; loss/FBt2/detection_loss"trainer/global_step2:JPÓ(£C°b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPL+ä|@±b; loss/FRt2/detection_loss"trainer/global_step2:JPÀz@²b; loss/FRt4/detection_loss"trainer/global_step2:JP­5(B³b= loss/SRqqt2/detection_loss"trainer/global_step2:JPègA´b< loss/FRt4/assignment_loss"trainer/global_step2:JP«BlAµb< loss/FBt2/assignment_loss"trainer/global_step2:JP ÖK”@¶b; loss/FBt4/detection_loss"trainer/global_step2:JP îÿ7·b2 loss/total_loss"trainer/global_step2:JPuš:A¸b< loss/FRt2/assignment_loss"trainer/global_step2:JPƒD[fC¹b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPSªP8Cºb> loss/SRqqt3/assignment_loss"trainer/global_step2:JPÄÒº@»b; loss/FRt3/detection_loss"trainer/global_step2:JPB:…A¼b< loss/FBt1/assignment_loss"trainer/global_step2:JP#ìÕB½b= loss/SRqqt4/detection_loss"trainer/global_step2:JPµè£@¾b; loss/FBt1/detection_loss"trainer/global_step2:JPŠU/.¿) _wandbruntime‚52  _runtime‚52¬-LÀÆ /loss/FBt3/assignment_loss‚2.686080657518827 /loss/FRt1/detection_loss‚3.1518555722146666 .loss/FBt3/detection_loss‚4.072090842211858 epoch‚0 trainer/global_step‚49 0loss/SRqqt3/detection_loss‚5.929625142237236 /loss/FRt1/assignment_loss‚7.454359225507052 /loss/FRt3/assignment_loss‚7.365003933509191 1loss/SRqqt1/assignment_loss‚5.504846489947775 0loss/FBt4/assignment_loss‚2.7221720218658447 1loss/SRqqt1/detection_loss‚1.9596360712919547 /loss/FBt2/detection_loss‚3.2513726250983255 _runtime‚ 52.095031844 1loss/SRqqt4/assignment_loss‚5.304460210066575 .loss/FRt2/detection_loss‚5.072042631070936 /loss/FRt4/detection_loss‚2.8612827983386113 0loss/SRqqt2/detection_loss‚4.769372076647622 /loss/FRt4/assignment_loss‚7.134518524385848 0loss/FBt2/assignment_loss‚2.7364913585574127 /loss/FBt4/detection_loss‚17.246221110224724 &loss/total_loss‚128.08521795750775 /loss/FRt2/assignment_loss‚7.249051200018989 1loss/SRqqt2/assignment_loss‚5.466087450299944 2loss/SRqqt3/assignment_loss‚5.4811751431432265 /loss/FRt3/detection_loss‚3.5849925074726343 _step‚3 $ _timestamp‚1.788428073739488e+09 /loss/FBt1/assignment_loss‚2.762072531382243 1loss/SRqqt4/detection_loss‚2.4581318366986054 /loss/FBt1/detection_loss‚11.862275997797648çÈj¤PÁÊ /loss/FBt4/detection_loss‚17.246221110224724 &loss/total_loss‚128.08521795750775 /loss/FRt2/assignment_loss‚7.249051200018989 1loss/SRqqt2/assignment_loss‚5.466087450299944 2loss/SRqqt3/assignment_loss‚5.4811751431432265 /loss/FRt3/detection_loss‚3.5849925074726343 _step‚3 $ _timestamp‚1.788428073739488e+09 /loss/FBt1/assignment_loss‚2.762072531382243 1loss/SRqqt4/detection_loss‚2.4581318366986054 /loss/FBt1/detection_loss‚11.862275997797648 /loss/FBt3/assignment_loss‚2.686080657518827 /loss/FRt1/detection_loss‚3.1518555722146666 .loss/FBt3/detection_loss‚4.072090842211858 epoch‚0 trainer/global_step‚49 0loss/SRqqt3/detection_loss‚5.929625142237236 /loss/FRt1/assignment_loss‚7.454359225507052 /loss/FRt3/assignment_loss‚7.365003933509191 1loss/SRqqt1/assignment_loss‚5.504846489947775 0loss/FBt4/assignment_loss‚2.7221720218658447 1loss/SRqqt1/detection_loss‚1.9596360712919547 /loss/FBt2/detection_loss‚3.2513726250983255 _runtime‚ 52.095031844 1loss/SRqqt4/assignment_loss‚5.304460210066575 .loss/FRt2/detection_loss‚5.072042631070936 /loss/FRt4/detection_loss‚2.8612827983386113 0loss/SRqqt2/detection_loss‚4.769372076647622 /loss/FRt4/assignment_loss‚7.134518524385848 0loss/FBt2/assignment_loss‚2.7364913585574127m¬ì“‚(:ˆ ±þäԎĵµ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.81843566894531* gpu.0.memoryAllocatedBytes‚ 29049815040 gpu.0.temp‚48.0 gpu.0.powerWatts‚284.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.9596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç¦4+Â(:  ±þäÔÞ„ô¶ disk.sda.out‚2.234375) disk./.usagePercent‚17.78755247540034# disk.nvme0n1.out‚207.4541015625 disk.rbd1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚4.078125 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125 disk.drbd2265.out‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 758.78125 network.sent‚90080 disk.sda2.in‚0 disk.sda.in‚0* proc.memory.percent‚18.024325370788574 disk.drbd2266.out‚0 disk.dm-0.in‚4.078125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2080.in‚0$ memory_percent‚95.36876678466797# proc.memory.rssMB‚ 2953.10546875 network.recv‚109326$ disk./.usageGB‚265.0473861694336 disk.dm-1.in‚0 disk.dm-0.out‚207.4541015625 disk.drbd2265.in‚0 disk.dm-1.out‚2.234375 disk.rbd1.out‚0 disk.drbd1716.out‚0 cpu‚25.355152355155845% disk.nvme0c0n1.out‚207.4541015625 disk.sda3.in‚0 disk.sda3.out‚2.234375/-ÿK“Ä‚(:ˆ ÀþäÔØ雵 gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚46.0 gpu.0.powerWatts‚165.697' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܧ“Ð4Å‚(:© ÀþäÔ’–· disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.078125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.078125& disk.nvme0c0n1.out‚283.30517578125 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.024373054504395 disk.dm-1.out‚ 2.55078125 disk.sda.in‚0 disk.drbd2265.out‚0# memory_percent‚95.3669548034668 network.sent‚140323# disk./.usageGB‚265.069637298584 disk.dm-1.in‚0 disk.drbd2266.in‚0! disk.dm-0.out‚283.30517578125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 2.55078125) disk./.usagePercent‚17.78904576734756 disk.dm-0.in‚4.078125 disk.sda3.out‚ 2.55078125 disk.rbd0.out‚0$ disk.nvme0n1.out‚283.30517578125# proc.memory.rssMB‚ 2953.11328125 cpu‚25.312098707123962 disk.rbd1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 network.recv‚116157 disk.sda2.out‚0& proc.memory.availableMB‚ 759.078125“÷ӞƂ(:“ ÏþäÔåõ·¶ gpu.0.gpu‚80.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚44.0 gpu.0.powerWatts‚145.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.080399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îúO†;Ç‚(:° ÏþäÔÎëܸ* disk./.usagePercent‚17.786019502285306 disk.sda1.in‚0 disk.drbd1716.out‚0 network.sent‚147394 disk.drbd2080.out‚0# disk.nvme0n1.out‚347.6123046875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.0859375 disk.sda3.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 866.91015625* proc.memory.percent‚18.024396896362305 disk.sda2.in‚0 disk.sda3.out‚ 3.3828125 disk.sda.out‚ 3.3828125$ memory_percent‚94.70880031585693" proc.memory.rssMB‚ 2953.1171875 disk.rbd0.in‚0 disk.dm-0.out‚347.6123046875 disk.sda1.out‚0 cpu‚25.277393304809777 disk.dm-1.out‚ 3.3828125 disk.drbd2266.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚347.6123046875 disk.drbd2080.in‚0 disk.sda.in‚0% disk./.usageGB‚265.02454376220703 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.recv‚121823 disk.drbd2266.in‚0 disk.dm-0.in‚4.078125 proc.cpu.threads‚17ü›Nà.È) _wandbruntime‚98  _runtime‚98ÏÀé3ÁÉ» & lr-AdamW/pg1‚1.675142451089607e-05 & lr-AdamW/pg2‚1.675142451089607e-05 trainer/global_step‚99 % _timestamp‚1.7884281201050704e+09 _runtime‚ 98.460091989 _step‚4~=FºÅÊ¿ & lr-AdamW/pg1‚1.675142451089607e-05 & lr-AdamW/pg2‚1.675142451089607e-05 trainer/global_step‚99 % _timestamp‚1.7884281201050704e+09 _runtime‚ 98.460091989 _step‚4iå…ß.Ë) _wandbruntime‚98  _runtime‚98pºËÌÂO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13403392300032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1737511 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 212474939392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 297192626688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3205336064 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 209370312704 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1796388 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 212118936666 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13400261281792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13190890969088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2129839 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 209015475290 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1029337 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3845742 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 trainer/global_step‚99 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108231 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108231 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13190890969088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13403392300032 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚767051 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚5 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1737511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13400261281792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1029340 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚767555 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11596811107328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 % _timestamp‚1.7884281201057734e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1796895 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11299566893568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2129839 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13189215699034 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1737517 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13190917360640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11596731385856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11299538759168 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12980200223744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3867356 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1737517 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 333375488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12980225651712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984ÔÕŽ¸Ì'ÍÆO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12980200223744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3867356 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1737517 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 333375488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12980225651712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13403392300032 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1737511 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 212474939392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 297192626688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3205336064 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 209370312704 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1796388 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 212118936666 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13400261281792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13190890969088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2129839 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 209015475290 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1029337 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3845742 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 trainer/global_step‚99 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108231 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108231 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13190890969088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13403392300032 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚767051 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚5 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1737511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13400261281792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1029340 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚767555 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11596811107328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 % _timestamp‚1.7884281201057734e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1796895 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11299566893568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2129839 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13189215699034 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1737517 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13190917360640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11596731385856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11299538759168¡à.Î) _wandbruntime‚99  _runtime‚99>kÒ&ÏÌM F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 333375488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13109597224960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2129321 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11712186090496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚774781 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11712104285184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 211124268390 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 % _timestamp‚1.7884281209987268e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 300154661888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1039593 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1814371 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 211482626560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 _step‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13322411448832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 214227729974 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2150955 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1754801 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13322385057280 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3905756 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3932160000 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13536998715392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13320721493350 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 214587266560 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2129321 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 211482626560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11411949623296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ’Ú:N&ÐÐM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 211482626560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11411949623296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 333375488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13109597224960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2129321 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11712186090496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚774781 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11712104285184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 211124268390 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 % _timestamp‚1.7884281209987268e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 300154661888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1039593 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1814371 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 211482626560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 _step‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13322411448832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 214227729974 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2150955 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1754801 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13322385057280 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3905756 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3932160000 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13536998715392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13320721493350 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 [JDeviceStatsMonitor.on_train_batch_en¬ ¹ˆd/allocated_bytes.small_pool.allocated‚ 214587266560 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2129321.ǧþ.Ñ) _wandbruntime‚99  _runtime‚99StdýGÒÁ 1loss/SRqqt2/detection_loss‚6.3169530619073795 _step‚7 0loss/FBt3/assignment_loss‚1.5885594257941613 1loss/SRqqt4/detection_loss‚2.6383829321111403 % _timestamp‚1.7884281210004144e+09 /loss/FRt2/assignment_loss‚7.223002489875345 1loss/SRqqt3/assignment_loss‚5.457868683338165 .loss/FBt3/detection_loss‚4.069338428286406 -loss/FBt4/detection_loss‚6.42126982831038 &loss/total_loss‚110.52284512896412 .loss/FRt2/detection_loss‚4.948948509552899 0loss/SRqqt1/detection_loss‚1.976449365980391 /loss/FBt1/detection_loss‚10.311899092267542 epoch‚0 _runtime‚ 99.355355206 /loss/FRt1/assignment_loss‚7.409527095158895 0loss/FBt1/assignment_loss‚1.9182983075871187 trainer/global_step‚99 0loss/SRqqt4/assignment_loss‚5.33989933998354 .loss/FRt1/detection_loss‚2.666220523168643 .loss/FRt3/detection_loss‚3.272680160231315 .loss/FBt2/detection_loss‚1.968934305585615 /loss/FRt4/assignment_loss‚7.200147939764935 1loss/SRqqt1/assignment_loss‚5.512692853382656 1loss/SRqqt2/assignment_loss‚5.454939630296495 /loss/FBt2/assignment_loss‚1.942405943967858 0loss/FBt4/assignment_loss‚1.7338621295415437 /loss/FRt4/detection_loss‚3.5509208648303603 0loss/SRqqt3/detection_loss‚4.293886530399322 /loss/FRt3/assignment_loss‚7.305757687642024 5¶@KÓÅ .loss/FRt1/detection_loss‚2.666220523168643 .loss/FRt3/detection_loss‚3.272680160231315 .loss/FBt2/detection_loss‚1.968934305585615 /loss/FRt4/assignment_loss‚7.200147939764935 1loss/SRqqt1/assignment_loss‚5.512692853382656 1loss/SRqqt2/assignment_loss‚5.454939630296495 /loss/FBt2/assignment_loss‚1.942405943967858 0loss/FBt4/assignment_loss‚1.7338621295415437 /loss/FRt4/detection_loss‚3.5509208648303603 0loss/SRqqt3/detection_loss‚4.293886530399322 /loss/FRt3/assignment_loss‚7.305757687642024 1loss/SRqqt2/detection_loss‚6.3169530619073795 _step‚7 0loss/FBt3/assignment_loss‚1.5885594257941613 1loss/SRqqt4/detection_loss‚2.6383829321111403 % _timestamp‚1.7884281210004144e+09 /loss/FRt2/assignment_loss‚7.223002489875345 1loss/SRqqt3/assignment_loss‚5.457868683338165 .loss/FBt3/detection_loss‚4.069338428286406 -loss/FBt4/detection_loss‚6.42126982831038 &loss/total_loss‚110.52284512896412 .loss/FRt2/detection_loss‚4.948948509552899 0loss/SRqqt1/detection_loss‚1.976449365980391 /loss/FBt1/detection_loss‚10.311899092267542 epoch‚0 _runtime‚ 99.355355206 /loss/FRt1/assignment_loss‚7.409527095158895 0loss/FBt1/assignment_loss‚1.9182983075871187 trainer/global_step‚99 0loss/SRqqt4/assignment_loss‚5.33989933998354Éy|WžÔ‚(:“ ÞþäÔ¡¼ß´ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.967' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.993399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZNé}>Õ‚(:³ ÞþäԦƪ· disk.nvme0c0n1.in‚4.078125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚4.078125& disk.nvme0c0n1.out‚398.38623046875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0$ memory_percent‚94.71004009246826 disk.drbd1716.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 866.70703125* proc.memory.percent‚18.024539947509766) disk./.usagePercent‚17.79167752479955 disk.drbd2266.in‚0 disk.sda3.out‚ 3.94140625 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2953.140625 network.recv‚126417 disk.dm-1.out‚ 3.94140625$ disk./.usageGB‚265.1088523864746 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 cpu‚25.249954226605723 proc.cpu.threads‚17 network.sent‚153783 disk.rbd1.out‚ 0.1484375 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚398.38623046875 disk.dm-1.in‚0 disk.dm-0.in‚4.078125! disk.dm-0.out‚398.38623046875 disk.sda.out‚ 3.94140625 disk.sda2.out‚0ç?}“Ö‚(:ˆ íþäÔ”õ´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈA;ׂ(:° íþäÔ£û¾¶* disk./.usagePercent‚17.792286566723508 disk.dm-0.in‚4.078125 disk.sda1.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 866.6484375 network.recv‚133350 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.1484375 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚94.71039772033691" disk.nvme0n1.out‚ 444.013671875 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 444.013671875 network.sent‚204346% disk./.usageGB‚265.11792755126953 disk.sda2.out‚0 disk.dm-1.out‚ 4.65234375 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 444.013671875 disk.sda1.out‚0 disk.nvme0n1.in‚4.078125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 4.65234375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 4.65234375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 cpu‚25.229674564291713 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2953.14453125* proc.memory.percent‚18.024563789367676ÒoŒ“Ø‚(:ˆ üþäÔ©éžµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚288.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cnDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20080595798016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2601717 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2601711 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3205336064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 314468721498 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2601711 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 315000419328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20083726816256 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3166083 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19765621770240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚149 % _timestamp‚1.7884281672507727e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19764519006042 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3187691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3166083 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16920081965568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19767647895386 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 317572182874 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2601717 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 445353163776 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3187691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16920110099968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19450050284544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19765621770240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17365463263744 _runtime‚ 145.606032757 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5767794 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 315000419328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1541990 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 318105046016 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1541987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20080595798016 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17365381445120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚9 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19765595378688 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 318105046016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 333375488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19450075712512 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 445299479552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19765595378688 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5789408 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20080595798016 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17365381445120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚9 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19765595378688 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 318105046016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 333375488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19450075712512 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 445299479552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19765595378688 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5789408 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20080595798016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2601717 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2601711 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3205336064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 314468721498 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2601711 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 315000419328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20083726816256 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3166083 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19765621770240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚149 % _timestamp‚1.7884281672507727e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19764519006042 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3187691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3166083 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16920081965568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19767647895386 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 317572182874 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2601717 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 445353163776 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3187691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16920110099968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19450050284544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19765621770240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17365463263744 _runtime‚ 145.606032757 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5767794 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 315000419328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1541990 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 318105046016 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1541987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187173 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5806168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.ÆzÕûÂlarge_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19899153689910 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 trainer/global_step‚149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3208807 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2618995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 448264225280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19896024800358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 317112733184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19579472713728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2710641 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3208807 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187173 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1158398 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 448317896192 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2619001 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1552240 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 _runtime‚ 146.503381738 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1157888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 320217373184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5827808 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2619001 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2710128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17480838860288 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3205336064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 333375488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2710641 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3208807 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187173 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1158398 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 448317896192 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2619001 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1552240 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 _runtime‚ 146.503381738 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1157888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 320217373184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5827808 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2619001 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2710128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17480838860288 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3205336064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 333375488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187173 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5806168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19899153689910 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 trainer/global_step‚149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3208807 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2618995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 448264225280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19896024800358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 317112733184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19579472713728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BdÆíÀ trainer/global_step‚199 % _timestamp‚1.7884282145013907e+09 _runtime‚ 192.856466567 _step‚12 ' lr-AdamW/pg1‚3.3672055329983005e-05 ' lr-AdamW/pg2‚3.3672055329983005e-05&NÖ¬ÊîÄ ' lr-AdamW/pg1‚3.3672055329983005e-05 ' lr-AdamW/pg2‚3.3672055329983005e-05 trainer/global_step‚199 % _timestamp‚1.7884282145013907e+09 _runtime‚ 192.856466567 _step‚12 ‰¤ñ0ï+ _wandbruntime‚192  _runtime‚192Õ®b¤À$ðÇO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26760930314240 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3465917 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4223935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26764061332480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26340326179840 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3589090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26342951202394 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 _runtime‚ 192.857425136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7689846 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26340299788288 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26760930314240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1533926 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79721472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25919925773312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23134057037824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22540653306368 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 593403731456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22540625171968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 593352144384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26339822313050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 420630525952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4245543 % _timestamp‚1.7884282145021589e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3588563 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 419921967706 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚13 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 420630525952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26340299788288 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7711460 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26764061332480 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4245543 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2054637 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3205336064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25919900345344 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3465911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3465917 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3465911 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚2313397jDœ¥ 7316352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 trainer/global_step‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 423025429082 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4223935 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 423735152640"¸ñÑ'ñËO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26764061332480 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4245543 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2054637 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3205336064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25919900345344 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3465911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3465917 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3465911 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23133977316352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 trainer/global_step‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 423025429082 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4223935 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 423735152640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26760930314240 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3465917 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4223935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26764061332480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26340326179840 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3589090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26342951202394 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 _runtime‚ 192.857425136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7689846 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26340299788288 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26760930314240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1533926 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79721472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25919925773312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23134057037824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22540653306368 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 593403731456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22540625171968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 593352144384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26339822313050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 420630525952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4245543 % _timestamp‚1.7884282145021589e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3588563 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 419921967706 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚13 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 420630525952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26340299788288 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7711460 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 \碚0ò+ _wandbruntime‚193  _runtime‚193Õ°ê@×&óÑM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1542087 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23249426855936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26471820268032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7728220 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26897667747840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3606471 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22653064170496 _runtime‚ 193.759593929 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4245025 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26897667747840 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4266659 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26471820268032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4245025 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26049297346560 trainer/global_step‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3205336064 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 596309014528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 % _timestamp‚1.7884282154040573e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1541581 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26471793876480 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7749860 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 422742839808 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26474456996918 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3483195 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 333375488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3932160000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 596362685440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26471328107366 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7749860 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4266659 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 425134222390 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 422030760806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7749860 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4266659 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 425134222390 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 422030760806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26471820268032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7728220 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26897667747840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3606471 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22653064170496 _runtime‚ 193.759593929 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4245025 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26897667747840 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4266659 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26471820268032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4245025 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26049297346560 trainer/global_step‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3205336064 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 596309014528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 % _timestamp‚1.7884282154040573e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1541581 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26471793876480 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7749860 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 422742839808 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26474456996918 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3483195 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 333375488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3932160000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 596362685440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26471328107366 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875&Ѽ_0õ+ _wandbruntime‚193  _runtime‚193`Wl“ü‚(:ˆ ÖÿäÔÆ£þ´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚46.0 gpu.0.powerWatts‚186.368' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷6@,ý‚(:¡ ÖÿäÔñ°Ž·* disk./.usagePercent‚17.804608209346494 disk.sda1.in‚0 disk.sda3.out‚8.15625$ memory_percent‚94.71406936645508& proc.memory.availableMB‚ 866.046875 cpu‚25.154305007674317 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.078125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0% disk./.usageGB‚265.30152893066406 disk.drbd2080.out‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚4.078125" proc.memory.rssMB‚ 2953.1640625* proc.memory.percent‚18.024682998657227 network.recv‚175374 disk.drbd2265.out‚0 network.sent‚339329 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.out‚ 800.7890625 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚8.15625 disk.drbd1716.out‚0 disk.sda.out‚8.15625 disk.sda2.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.nvme0n1.out‚ 800.7890625 disk.dm-0.in‚4.078125 disk.dm-0.out‚ 800.7890625ôM ‡“þ‚(:ˆ åÿäÔãÆð´ gpu.0.gpu‚87.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#y7=ÿ‚(:² åÿäÔ÷㺷 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 8.79296875 disk.drbd2265.in‚0 network.sent‚346025 disk.dm-1.in‚0 disk.dm-1.out‚ 8.79296875 disk.drbd2266.in‚0 disk.dm-0.out‚843.1962890625 disk.drbd2080.out‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2953.1640625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 network.recv‚180765% disk./.usageGB‚265.31324005126953 disk.sda3.in‚0 disk.sda3.out‚ 8.79296875 disk.nvme0n1.in‚4.078125# disk.nvme0n1.out‚843.1962890625 disk.sda2.out‚0* disk./.usagePercent‚17.805394152476612 disk.drbd2080.in‚0 disk.dm-0.in‚4.078125% proc.memory.availableMB‚ 866.28125* proc.memory.percent‚18.024682998657227 cpu‚25.148350744955394 disk.sda2.in‚0 disk.nvme0c0n1.in‚4.078125 disk.sda1.in‚0% disk.nvme0c0n1.out‚843.1962890625 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚94.71263885498047 proc.cpu.threads‚17'¦™0€+ _wandbruntime‚240  _runtime‚240û¦£8ÆÀ % _timestamp‚1.7884282618879473e+09 _runtime‚ 240.242981493 _step‚16 ' lr-AdamW/pg1‚4.2132370739526477e-05 ' lr-AdamW/pg2‚4.2132370739526477e-05 trainer/global_step‚249¥›˜Ê‚Ä _runtime‚ 240.242981493 _step‚16 ' lr-AdamW/pg1‚4.2132370739526477e-05 ' lr-AdamW/pg2‚4.2132370739526477e-05 trainer/global_step‚249 % _timestamp‚1.7884282618879473e+09ˆ»+0ƒ+ _wandbruntime‚240  _runtime‚240³2ìÍ'„ÇO _runtime‚ 240.244264034 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3220016128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 525375213914 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9611898 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9633512 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1916072 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28902367042048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2567290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 _step‚17 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32915004197888 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 526260632576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 741252347904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 526260632576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33444395848704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4330117 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5281787 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4330117 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2567287 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33441264830464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28161196512768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32915030589440 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32915125620058 % _timestamp‚1.7884282618888166e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33444395848704 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33441264830464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4330111 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32389750406144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1916607 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 trainer/global_step‚249 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 529365259264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28902448860672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4483359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32918254509402 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5281787 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32915030589440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32915004197888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4483897 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28161168378368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 741198663680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 529365259264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4330111 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5303395 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Ÿ~tœÑ'…ËO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32915030589440 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32915125620058 % _timestamp‚1.7884282618888166e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33444395848704 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33441264830464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4330111 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32389750406144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1916607 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 trainer/global_step‚249 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 529365259264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28902448860672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4483359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32918254509402 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5281787 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32915030589440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32915004197888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4483897 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28161168378368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 741198663680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 529365259264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4330111 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5303395 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 240.244264034 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3220016128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 525375213914 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9611898 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9633512 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1916072 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28902367042048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2567290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 _step‚17 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32915004197888 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 526260632576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 741252347904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 526260632576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33444395848704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4330117 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5281787 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4330117 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2567287 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33441264830464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28161196512768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ö+œ0†+ _wandbruntime‚241  _runtime‚241G‘·‡ÑM QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4347401 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29017743360000 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3220016128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33046631414374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33574871232512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33046524677632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9671912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 744217788416 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4347401 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33578002264064 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4501240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33046524677632 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9671912 trainer/global_step‚249 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5302877 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528372946432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33578002264064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚18 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 744164117504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4347395 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33046498286080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 528372946432 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3932160000 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33049760303926 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2577543 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32519172835328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1923700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 531477586432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 530587468598 % _timestamp‚1.7884282627983367e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 527484007014 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4501763 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5324511 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1924220 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28273607376896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3932160000 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33049760303926 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2577543 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32519172835328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1923700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 531477586432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 530587468598 % _timestamp‚1.7884282627983367e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 527484007014 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4501763 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5324511 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1924220 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28273607376896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33046631414374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33574871232512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33046524677632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9671912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 744217788416 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4347401 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33578002264064 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4501240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33046524677632 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9671912 trainer/global_step‚249 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5302877 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528372946432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33578002264064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚18 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 744164117504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4347395 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33046498286080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 528372946432 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875&J/¤0‰+ _wandbruntime‚241  _runtime‚241Ë.K\NŠÈ 0loss/FBt4/assignment_loss‚0.6060535061776384 /loss/FRt3/assignment_loss‚7.280709033109704 0loss/FRt4/assignment_loss‚7.2204901071695184 .loss/FBt2/detection_loss‚4.029028322015906 /loss/FBt3/detection_loss‚2.2943686177661897 _runtime‚ 241.15487175 1loss/SRqqt1/assignment_loss‚3.692328924356505 2loss/SRqqt3/assignment_loss‚3.5071191692352297 -loss/FBt1/detection_loss‚9.93710111344562 -loss/FBt4/detection_loss‚7.34616687216542 1loss/FBt3/assignment_loss‚0.17633445908580267 1loss/SRqqt3/detection_loss‚3.2281038761138916 1loss/SRqqt4/detection_loss‚1.7104184059282779 trainer/global_step‚249 2loss/SRqqt4/assignment_loss‚3.3011421466219253 0loss/FBt2/assignment_loss‚0.8152693213362779 -loss/FRt1/detection_loss‚4.88991153108723 /loss/FRt2/detection_loss‚3.2520599961280823 1loss/SRqqt2/detection_loss‚3.8791656385768545 %loss/total_loss‚93.21159094739129 _step‚19 % _timestamp‚1.7884282627999358e+09 .loss/FRt2/assignment_loss‚6.97528009944492 2loss/SRqqt2/assignment_loss‚3.5917690125378696 0loss/FBt1/assignment_loss‚0.7030585585162044 epoch‚0 .loss/FRt4/detection_loss‚2.888904714383758 /loss/FRt1/assignment_loss‚7.392429520102108 .loss/FRt3/detection_loss‚3.315654448670696 1loss/SRqqt1/detection_loss‚1.1787235534156626nõ¤œR‹Ì 2loss/SRqqt4/assignment_loss‚3.3011421466219253 0loss/FBt2/assignment_loss‚0.8152693213362779 -loss/FRt1/detection_loss‚4.88991153108723 /loss/FRt2/detection_loss‚3.2520599961280823 1loss/SRqqt2/detection_loss‚3.8791656385768545 %loss/total_loss‚93.21159094739129 _step‚19 % _timestamp‚1.7884282627999358e+09 .loss/FRt2/assignment_loss‚6.97528009944492 2loss/SRqqt2/assignment_loss‚3.5917690125378696 0loss/FBt1/assignment_loss‚0.7030585585162044 epoch‚0 .loss/FRt4/detection_loss‚2.888904714383758 /loss/FRt1/assignment_loss‚7.392429520102108 .loss/FRt3/detection_loss‚3.315654448670696 1loss/SRqqt1/detection_loss‚1.1787235534156626 0loss/FBt4/assignment_loss‚0.6060535061776384 /loss/FRt3/assignment_loss‚7.280709033109704 0loss/FRt4/assignment_loss‚7.2204901071695184 .loss/FBt2/detection_loss‚4.029028322015906 /loss/FBt3/detection_loss‚2.2943686177661897 _runtime‚ 241.15487175 1loss/SRqqt1/assignment_loss‚3.692328924356505 2loss/SRqqt3/assignment_loss‚3.5071191692352297 -loss/FBt1/detection_loss‚9.93710111344562 -loss/FBt4/detection_loss‚7.34616687216542 1loss/FBt3/assignment_loss‚0.17633445908580267 1loss/SRqqt3/detection_loss‚3.2281038761138916 1loss/SRqqt4/detection_loss‚1.7104184059282779 trainer/global_step‚2499Ø Ä’Œ‚(:‡ ôÿäÔ¤‡€¶ gpu.0.gpu‚67.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚46.0 gpu.0.powerWatts‚131.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬||3‚(:¨ ôÿäÔ¶Þµ·! disk.nvme0n1.out‚ 894.74609375 disk.dm-1.out‚ 9.1640625 disk.dm-0.in‚4.078125 disk.sda1.in‚0 disk.drbd2080.out‚0 network.sent‚351780 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 9.1640625 disk.drbd2265.in‚0 proc.cpu.threads‚17% disk./.usageGB‚265.32372665405273# disk.nvme0c0n1.out‚ 894.74609375 disk.sda3.in‚0 disk.sda3.out‚ 9.1640625 disk.drbd1716.in‚0$ memory_percent‚94.71473693847656 disk.nvme0n1.in‚4.078125 disk.nvme0c0n1.in‚4.078125 disk.dm-0.out‚ 894.74609375$ proc.memory.availableMB‚865.9375# proc.memory.rssMB‚ 2953.16796875 cpu‚25.143101780341016) disk./.usagePercent‚17.80609791718824 disk.sda.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.024706840515137 network.recv‚184964 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0šÒKSžŽ‚(:“ ƒ€åÔ똀µ gpu.0.gpu‚94.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚273.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.635400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÃXõ;‚(:° ƒ€åÔ÷íï· proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 0.306640625 disk.nvme0c0n1.in‚4.078125% disk.nvme0c0n1.out‚941.0185546875 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 network.sent‚401701 disk.dm-1.out‚ 9.7734375 disk.dm-0.in‚4.078125 disk.sda1.in‚0$ memory_percent‚94.71499919891357 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2953.1796875% disk./.usageGB‚265.33423614501953 disk.drbd1716.in‚0) disk./.usagePercent‚17.80680321794507 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.out‚ 9.7734375 network.recv‚190790# disk.nvme0n1.out‚941.0185546875 disk.rbd1.in‚0 disk.dm-0.out‚941.0185546875 disk.sda3.out‚ 9.7734375 disk.drbd2080.in‚0 disk.sda.in‚0* proc.memory.percent‚18.024778366088867 disk.drbd2266.out‚0( proc.memory.availableMB‚ 865.89453125 cpu‚25.13745500071681 _!ž‚(:“ ’€åÔÚñе gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚306.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.364200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²…˜J8‘‚(:­ ’€åÔùçÓ· disk.sda1.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0$ memory_percent‚94.71583366394043 proc.cpu.threads‚17 disk.rbd1.out‚ 0.306640625 disk.dm-0.in‚4.078125 disk.sda3.out‚10.15625 disk.drbd2080.in‚0( disk./.usagePercent‚17.8073193291341 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚10.15625 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 network.sent‚407852% disk./.usageGB‚265.34192657470703 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2953.18359375 disk.dm-0.out‚997.0986328125 disk.sda2.out‚0 disk.dm-1.out‚10.15625% disk.nvme0c0n1.out‚997.0986328125# disk.nvme0n1.out‚997.0986328125* proc.memory.percent‚18.024802207946777 cpu‚25.133268340313236 network.recv‚195211 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.sda1.out‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 865.7578125÷u—È0’+ _wandbruntime‚287  _runtime‚287ÿ1Õ•Ä“¾ & lr-AdamW/pg2‚5.059268614906995e-05 trainer/global_step‚299 % _timestamp‚1.7884283092266061e+09 _runtime‚ 287.581677086 _step‚20 & lr-AdamW/pg1‚5.059268614906995e-05óã4¶È” & lr-AdamW/pg1‚5.059268614906995e-05 & lr-AdamW/pg2‚5.059268614906995e-05 trainer/global_step‚299 % _timestamp‚1.7884283092266061e+09 _runtime‚ 287.581677086 _step‚20C‰ 0•+ _wandbruntime‚287  _runtime‚287Š¼Ke&–ËO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11555564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2299381 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38859625894912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40121599346688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3079940 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39490428927066 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 631890739200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33781711584768 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 889172837888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39489708607488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39489708607488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33781739719168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2298855 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 889117056512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39489734999040 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11533950 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6339639 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40124730364928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 634995365888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚21 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 % _timestamp‚1.7884283092273357e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 633931921498 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6339639 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6339639 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40124730364928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 634995365888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚21 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 % _timestamp‚1.7884283092273357e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 633931921498 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6339639 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40121599346688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3079940 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0È¿o40˜+ _wandbruntime‚288  _runtime‚288r©º5Ú&™ÔM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39621229087232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11572324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39625063610934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6382363 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5396649 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40258336780288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 632937253222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5211601 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33894122448896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39621229087232 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 637107693056 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5211601 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6360729 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 892125203456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11572324 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39621202695680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3090193 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39621934721382 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 636040714806 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11593964 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34786275786752 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39621202695680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40258336780288 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 _runtime‚ 288.47330558 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6360729 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5211595 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2306459 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34786193981440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33894150583296 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11593964 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701EŽ"Þ&šØM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6360729 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5211595 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2306459 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34786193981440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33894150583296 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11593964 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39621229087232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11572324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39625063610934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6382363 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5396649 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40258336780288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 632937253222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5211601 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33894122448896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39621229087232 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 637107693056 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5211601 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6360729 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 892125203456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11572324 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39621202695680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3090193 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39621934721382 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 636040714806 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11593964 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34786275786752 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39621202695680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40258336780288 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 _runtime‚ 288.47330558g¢ä0›+ _wandbruntime‚288  _runtime‚288|róNœÈ 1loss/SRqqt1/assignment_loss‚3.512342438553319 2loss/SRqqt4/assignment_loss‚3.2499693285060833 0loss/FBt3/assignment_loss‚0.3277281153448192 1loss/SRqqt4/detection_loss‚1.2276815576640936 .loss/FRt3/assignment_loss‚7.37890112059457 /loss/FBt2/assignment_loss‚0.652142977565527 _runtime‚ 288.474457645 _step‚23 /loss/FRt4/assignment_loss‚7.188481533781011 .loss/FRt2/detection_loss‚4.396674843061538 -loss/FBt1/detection_loss‚5.75389553092677 /loss/FBt3/detection_loss‚3.0637636950045515 % _timestamp‚1.7884283101194708e+09 2loss/SRqqt3/assignment_loss‚3.3836931185288863 0loss/FBt1/assignment_loss‚0.4381245303124286 0loss/FBt4/assignment_loss‚0.7568295177506198 .loss/FRt1/detection_loss‚3.768041658401489 %loss/total_loss‚86.48761157660007 /loss/FRt2/assignment_loss‚7.209434009733654 1loss/SRqqt2/detection_loss‚3.6547611890868708 .loss/FBt4/detection_loss‚7.272578769786791 epoch‚0 trainer/global_step‚299 /loss/FRt1/assignment_loss‚7.330167717403835 2loss/SRqqt2/assignment_loss‚3.5966328274119985 /loss/FRt3/detection_loss‚2.3321804394147225 .loss/FRt4/detection_loss‚3.342340878428931 0loss/SRqqt1/detection_loss‚3.545124856670472 1loss/SRqqt3/detection_loss‚2.6582320622422477 /loss/FBt2/detection_loss‚0.4478888604248442‘nPA!Ì % _timestamp‚1.7884283101194708e+09 2loss/SRqqt3/assignment_loss‚3.3836931185288863 0loss/FBt1/assignment_loss‚0.4381245303124286 0loss/FBt4/assignment_loss‚0.7568295177506198 .loss/FRt1/detection_loss‚3.768041658401489 %loss/total_loss‚86.48761157660007 /loss/FRt2/assignment_loss‚7.209434009733654 1loss/SRqqt2/detection_loss‚3.6547611890868708 .loss/FBt4/detection_loss‚7.272578769786791 epoch‚0 trainer/global_step‚299 /loss/FRt1/assignment_loss‚7.330167717403835 2loss/SRqqt2/assignment_loss‚3.5966328274119985 /loss/FRt3/detection_loss‚2.3321804394147225 .loss/FRt4/detection_loss‚3.342340878428931 0loss/SRqqt1/detection_loss‚3.545124856670472 1loss/SRqqt3/detection_loss‚2.6582320622422477 /loss/FBt2/detection_loss—¾ÖO1‚0.4478888604248442 1loss/SRqqt1/assignment_loss‚3.512342438553319 2loss/SRqqt4/assignment_loss‚3.2499693285060833 0loss/FBt3/assignment_loss‚0.3277281153448192 1loss/SRqqt4/detection_loss‚1.2276815576640936 .loss/FRt3/assignment_loss‚7.37890112059457 /loss/FBt2/assignment_loss‚0.652142977565527 _runtime‚ 288.474457645 _step‚23 /loss/FRt4/assignment_loss‚7.188481533781011 .loss/FRt2/detection_loss‚4.396674843061538 -loss/FBt1/detection_loss‚5.75389553092677 /loss/FBt3/detection_loss‚3.0637636950045515tîÌ“ž‚(:ˆ ¡€åÔæò¡µ gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•6/EŸ‚(:º ¡€åÔÌÈÿ¶ disk.dm-0.in‚4.078125 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚413557 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.306640625 disk.drbd1716.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125! disk.dm-0.out‚1040.2080078125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 disk.sda2.in‚0# proc.memory.rssMB‚ 2953.18359375* disk./.usagePercent‚17.807842352526553 disk.drbd1716.in‚0 disk.sda.out‚ 10.76171875$ memory_percent‚94.71800327301025 cpu‚25.12954163884724 disk.sda3.out‚ 10.76171875 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0& disk.nvme0c0n1.out‚1040.2080078125 network.recv‚199533 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 10.76171875( proc.memory.availableMB‚ 865.40234375$ disk./.usageGB‚265.3497200012207$ disk.nvme0n1.out‚1040.2080078125 disk.sda2.out‚0 disk.dm-1.in‚0* proc.memory.percent‚18.024802207946777D  ‚(:’ °€åԉ笵 gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚49.0 gpu.0.powerWatts‚285.334' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.06679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ0WÁI¡‚(:¾ °€åÔ¡È’·' disk.nvme0c0n1.out‚1091.70556640625 disk.dm-0.in‚4.078125 disk.sda.in‚0 disk.nvme0n1.in‚4.078125 disk.drbd2266.out‚0 disk.drbd2266.in‚0$ memory_percent‚94.71662044525146( proc.memory.availableMB‚ 865.62890625 disk.rbd0.out‚0 disk.sda3.out‚ 11.17578125 network.recv‚204331 disk.rbd0.in‚0# proc.memory.rssMB‚ 2953.18359375* proc.memory.percent‚18.024802207946777 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 11.17578125 cpu‚25.125281662521704 network.sent‚462917) disk./.usagePercent‚17.80869101750256% disk.nvme0n1.out‚1091.70556640625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 11.17578125 disk.dm-1.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚1091.70556640625 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda3.in‚0% disk./.usageGB‚265.36236572265625 disk.rbd1.out‚ 0.306640625¼1p“¢‚(:ˆ ¿€åÔ™‘Ó´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚286.597' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„yo‰@£‚(:µ ¿€åÔ÷â„· disk.sda.out‚ 11.7578125 disk.rbd0.out‚0 disk.rbd1.out‚ 0.306640625% disk./.usageGB‚265.37210845947266 disk.dm-0.in‚4.078125 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 11.7578125* disk./.usagePercent‚17.809344860745007 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚1134.62353515625" proc.memory.availableMB‚866.75* proc.memory.percent‚18.024802207946777 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2953.18359375 disk.sda3.out‚ 11.7578125 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0% disk.nvme0n1.out‚1134.62353515625 network.sent‚469321 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125$ memory_percent‚94.70977783203125 cpu‚25.12212482953621 disk.drbd2266.out‚0 network.recv‚208921" disk.dm-0.out‚1134.62353515625 disk.sda.in‚0ÞÏ_r0¤+ _wandbruntime‚334  _runtime‚334ã]"GÄ¥¾ trainer/global_step‚349 % _timestamp‚1.7884283565690212e+09 _runtime‚ 334.924061921 _step‚24 & lr-AdamW/pg1‚5.905300155861342e-05 & lr-AdamW/pg2‚5.905300155861342e-05잮Ȧ _runtime‚ 334.924061921 _step‚24 & lr-AdamW/pg1‚5.905300155861342e-05 & lr-AdamW/pg2‚5.905300155861342e-05 trainer/global_step‚349 % _timestamp‚1.7884283565690212e+09qÒÔÉ0§+ _wandbruntime‚334  _runtime‚334->‘Ó'¨ÍO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058511 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2678796 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46068861123418 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79721472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39402254791168 _step‚25 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46805064881152 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 737520845824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3592590 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40439815724544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7419099 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 trainer/global_step‚349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46064413017088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 740625472512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7397491 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46801933862912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 739385167706 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13477616 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46805064881152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058517 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6270864 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 736281706330ãžà„×'©ÑO OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058511 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2678796 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51587072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46068861123418 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79721472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39402254791168 _step‚25 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46805064881152 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 737520845824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3592590 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40439815724544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7419099 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 trainer/global_step‚349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46064413017088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 740625472512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7397491 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46801933862912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 739385167706 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13477616 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46805064881152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058517 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6270864 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 736281706330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688¥÷ƒ0ª+ _wandbruntime‚335  _runtime‚335ün¯¢«ÖM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13516016 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7418581 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6075801 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51573760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46935540264960 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40555201531904 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46195933496832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46195907105280 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6075801 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2686309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6075795 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 742737799680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚349 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6075795 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7440215 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7440215 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2685814 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39514665655296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45458847528960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46195933496832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚79708160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788428357468994e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1040456168448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40555121823744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7418581 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39514693789696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 741493961014 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13516016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 739633159680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6288654 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3602840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13516016 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7418581 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6075801 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51573760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46935540264960 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40555201531904 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46195933496832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46195907105280 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6075801 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2686309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6075795 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 742737799680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚349 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6075795 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7440215 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7440215 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2685814 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39514665655296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45458847528960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46195933496832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚79708160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3177 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788428357468994e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1040456168448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40555121823744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7418581 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39514693789696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 741493961014 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13516016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 739633159680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6288654 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3602840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 B³‚(:³ Ý€åÔ“„á· disk.nvme0c0n1.in‚4.078125$ memory_percent‚94.71025466918945 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.078125& disk.nvme0c0n1.out‚1234.2158203125 network.recv‚219384 disk.sda3.out‚ 12.9609375$ disk.nvme0n1.out‚1234.2158203125 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0 network.sent‚525777% disk./.usageGB‚265.37256622314453 disk.sda1.in‚0 disk.drbd2080.in‚0& proc.memory.availableMB‚ 866.671875 disk.rbd0.in‚0 disk.drbd2266.out‚0! disk.dm-0.out‚1234.2158203125 cpu‚25.117065810090978* disk./.usagePercent‚17.809375581649117 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚4.078125* proc.memory.percent‚18.024826049804688 disk.sda.out‚ 12.9609375 disk.drbd1716.out‚0 disk.drbd2080.out‚0 proc.memory.rssMB‚ 2953.1875 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 12.9609375ÉnŽž´‚(:“ ì€åÔ“€¼µ gpu.0.gpu‚72.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚47.0 gpu.0.powerWatts‚139.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.821800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÔž§1µ‚(:¦ ì€åÔÒ†­·* disk./.usagePercent‚17.809848171557324$ disk./.usageGB‚265.3796081542969 disk.drbd2080.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚1307.6533203125 disk.drbd2266.in‚0 cpu‚25.114607492836903 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.sda3.out‚13.375 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚13.375 proc.memory.rssMB‚ 2953.1875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0* proc.memory.percent‚18.024826049804688 network.recv‚223083 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0& proc.memory.availableMB‚ 866.671875 disk.drbd2265.in‚0 disk.sda2.in‚0$ memory_percent‚94.71025466918945& disk.nvme0c0n1.out‚1307.6533203125 disk.dm-0.in‚4.078125 proc.cpu.threads‚17 network.sent‚531332! disk.dm-0.out‚1307.6533203125 disk.nvme0n1.in‚4.078125 disk.dm-1.out‚13.375”œ±$0¶+ _wandbruntime‚382  _runtime‚382"+êÄ·¾ & lr-AdamW/pg1‚6.751331696815688e-05 & lr-AdamW/pg2‚6.751331696815688e-05 trainer/global_step‚399 % _timestamp‚1.7884284038796082e+09 _runtime‚ 382.234640954 _step‚289 ǖȸ _runtime‚ 382.234640954 _step‚28 & lr-AdamW/pg1‚6.751331696815688e-05 & lr-AdamW/pg2‚6.751331696815688e-05 trainer/global_step‚399 % _timestamp‚1.7884284038796082e+09Dý…0¹+ _wandbruntime‚382  _runtime‚382 \¹Ó'ºÍO YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 841734952538 % _timestamp‚1.7884284038803096e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52644164430426 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846255579136 trainer/global_step‚399 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 846255579136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6922711 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53482268379136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7166911 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 844838413914 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1185330458624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15399668 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53485399397376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8455343 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 843150952448 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6922717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4105240 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53485399397376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52639143818240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52639117426688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3061132 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1185386240000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46208212371968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4105237 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3224210432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚29 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8476951 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52639117426688 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51799300588544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 382.235574506 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53482268379136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52641035541082 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6922711 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6922717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3061671 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46208128456192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45022797997568 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51799326016512 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15378054 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8455343 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0½ü´Ï»ÑO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52639117426688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3061132 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1185386240000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 O?Deï7Ï"viceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46208212371968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4105237 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3224210432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚29 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8476951 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52639117426688 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51799300588544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 382.235574506 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53482268379136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52641035541082 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6922711 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6922717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3061671 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46208128456192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45022797997568 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51799326016512 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15378054 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8455343 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 841734952538 % _timestamp‚1.7884284038803096e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52644164430426 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846255579136 trainer/global_step‚399 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 846255579136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6922711 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53482268379136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7166911 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 844838413914 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1185330458624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15399668 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53485399397376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8455343 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 843150952448 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6922717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1875 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4105240 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53485399397376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52639143818240:mŒ«0¼+ _wandbruntime‚383  _runtime‚383"©•>Ý&½×M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4115490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52770611514880 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 845263266304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 _runtime‚ 383.128608873 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15438068 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 845263266304 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1188271743488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51928723017728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15416428 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52775670224950 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8498067 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8498067 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45135208861696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 848367906304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52770611514880 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8476433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3069189 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7184161 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45135236996096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188327511552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52772541335398 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6939995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3068671 _step‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46323480605184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884284047729762e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15416428 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 843843745638 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15438068 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 trainer/global_step‚399 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8498067 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8498067 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45135208861696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 848367906304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52770611514880 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8476433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3069189 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7184161 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45135236996096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188327511552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52772541335398 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6939995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3068671 _step‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46323480605184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884284047729762e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15416428 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 843843745638 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15438068 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28504489984 trainer/global_step‚399 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4115490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52770611514880 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 845263266304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 _runtime‚ 383.128608873 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15438068 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 845263266304 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634•p0¿+ _wandbruntime‚383  _runtime‚383ïñØOÀÉ /loss/FRt3/detection_loss‚3.5040161363237226 % _timestamp‚1.7884284047750387e+09 /loss/FRt2/assignment_loss‚7.233910904990302 2loss/SRqqt3/assignment_loss‚2.5945552549664934 1loss/FBt4/assignment_loss‚0.36388609348270823 0loss/FRt4/assignment_loss‚7.0907363398321746 1loss/SRqqt2/assignment_loss‚3.389419289735647 2loss/SRqqt4/assignment_loss‚3.1282101251460888 /loss/FRt1/assignment_loss‚7.403035698867425 1loss/SRqqt1/assignment_loss‚3.243701485070315 /loss/FRt1/detection_loss‚3.9301132348979393 0loss/SRqqt1/detection_loss‚3.564262375955216 -loss/FBt2/detection_loss‚2.98850356548376 _step‚31 .loss/FBt1/detection_loss‚2.462408442776147 %loss/total_loss‚89.47090581888534 epoch‚0 _runtime‚ 383.129992354 1loss/FBt1/assignment_loss‚0.24902892214110275 .loss/FRt2/detection_loss‚5.149751689698961 0loss/SRqqt2/detection_loss‚6.223338672747979 1loss/SRqqt3/detection_loss‚2.4358303820093474 trainer/global_step‚399 .loss/FRt4/detection_loss‚2.137562133243372 0loss/SRqqt4/detection_loss‚1.768401502748921 .loss/FBt3/detection_loss‚7.493826253073556 .loss/FBt4/detection_loss‚4.246229290548298 /loss/FRt3/assignment_loss‚7.373580267031987 0loss/FBt2/assignment_loss‚0.9774300950978484 0loss/FBt3/assignment_loss‚0.5191676630160406»-‹ SÁÍ .loss/FBt4/detection_loss‚4.246229290548298 /loss/FRt3/assignment_loss‚7.373580267031987 0loss/FBt2/assignment_loss‚0.9774300950978484 0loss/FBt3/assignment_loss‚0.5191676630160406 /loss/FRt3/detection_loss‚3.5040161363237226 % _timestamp‚1.7884284047750387e+09 /loss/FRt2/assignment_loss‚7.233910904990302 2loss/SRqqt3/assignment_loss‚2.5945552549664934 1loss/FBt4/assignment_loss‚0.36388609348270823 0loss/FRt4/assignment_loss‚7.0907363398321746 1loss/SRqqt2/assignment_loss‚3.389419289735647 2loss/SRqqt4/assignment_loss‚3.1282101251460888 /loss/FRt1/assignment_loss‚7.403035698867425 1loss/SRqqt1/assignment_loss‚3.243701485070315 /loss/FRt1/detection_loss‚3.9301132348979393 0loss/SRqqt1/detection_loss‚3.564262375955216 -loss/FBt2/detection_loss‚2.98850356548376 _step‚31 .loss/FBt1/detection_loss‚2.462408442776147 %loss/total_loss‚89.47090581888534 epoch‚0 _runtime‚ 383.129992354 1loss/FBt1/assignment_loss‚0.24902892214110275 .loss/FRt2/detection_loss‚5.149751689698961 0loss/SRqqt2/detection_loss‚6.223338672747979 1loss/SRqqt3/detection_loss‚2.4358303820093474 trainer/global_step‚399 .loss/FRt4/detection_loss‚2.137562133243372 0loss/SRqqt4/detection_loss‚1.768401502748921 .loss/FBt3/detection_loss‚7.493826253073556ŒÛRžÂ‚(:“ û€åÔ Þ¹µ gpu.0.gpu‚66.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚276.746' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.349199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A^Þc[Â(:° û€åԡˆ· cpu‚25.11231212331784 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 13.953125 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 13.953125$ memory_percent‚94.71213817596436 network.sent‚537668 disk.sda.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 13.953125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚4.078125& disk.nvme0c0n1.out‚1349.8486328125 disk.dm-0.in‚4.078125 disk.sda3.in‚0$ disk./.usageGB‚265.3836479187012 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0! disk.dm-0.out‚1349.8486328125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda1.in‚0* disk./.usagePercent‚17.810119283536086 disk.drbd2080.in‚0$ disk.nvme0n1.out‚1349.84863281OCŠà25 disk.sda2.out‚0 proc.memory.rssMB‚ 2953.1875* proc.memory.percent‚18.024826049804688 network.recv‚228180 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚4.078125( proc.memory.availableMB‚ 866.36328125çnKÖ‘Ä‚(:† ŠåÔŠ¦µ gpu.0.gpu‚48.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚48.0 gpu.0.powerWatts‚129.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø/*¼<Å‚(:± ŠåÔâß×· disk.dm-1.out‚ 14.36328125 disk.drbd2266.out‚0$ proc.memory.availableMB‚866.8125 disk.sda3.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚1380.7548828125 disk.rbd1.out‚ 0.306640625& disk.nvme0c0n1.out‚1380.7548828125 disk.sda.out‚ 14.36328125 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚4.078125 proc.memory.rssMB‚ 2953.1875 network.sent‚587543 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 14.36328125 disk.drbd2265.in‚0 disk.dm-0.in‚4.078125* proc.memory.percent‚18.024826049804688 proc.cpu.threads‚17) disk./.usagePercent‚17.81110363250524 disk.sda1.out‚0 disk.drbd2080.out‚0! disk.dm-0.out‚1380.7548828125$ memory_percent‚94.70939636230469 cpu‚25.11006877269183 network.recv‚233651$ disk./.usageGB‚265.3983154296875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0xOEJ“Æ‚(:ˆ ™åԯ܉µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±<ÓDÇ‚(:¹ ™åÔª·Í¶$ memory_percent‚94.71089839935303( proc.memory.availableMB‚ 866.56640625 network.sent‚594154 disk.sda3.out‚ 15.21484375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2953.1953125 cpu‚25.10817330394932 disk.nvme0c0n1.in‚4.078125* disk./.usagePercent‚17.806264834100563 disk.dm-1.out‚ 15.21484375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.024873733520508! disk.dm-0.out‚1426.0478515625 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.078125$ disk./.usageGB‚265.3262138366699 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.recv‚238904& disk.nvme0c0n1.out‚1426.0478515625$ disk.nvme0n1.out‚1426.0478515625 disk.dm-1.in‚0 disk.sda.out‚ 15.21484375 disk.nvme0n1.in‚4.078125‡3sn0È+ _wandbruntime‚429  _runtime‚429úãžžÄɾ _runtime‚ 429.654267258 _step‚32 & lr-AdamW/pg1‚7.597363237770036e-05 & lr-AdamW/pg2‚7.597363237770036e-05 trainer/global_step‚449 % _timestamp‚1.7884284512992518e+09W¨ÊÈÊ & lr-AdamW/pg1‚7.597363237770036e-05 & lr-AdamW/pg2‚7.597363237770036e-05 trainer/global_step‚449 % _timestamp‚1.7884284512992518e+09 _runtime‚ 429.654267258 _step‚32 (.øÕ0Ë+ _wandbruntime‚429  _runtime‚429¹riÓ'ÌÍO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9534803 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60165733913600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947188198746 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617890 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51976315900416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59219467737434 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59216338848090 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9513195 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3443567 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3443007 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59213821836288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59213821836288 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50643369338368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8060894 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 950291660122 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 _step‚33 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3224210432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51976399816192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50643341203968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884284513000593e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60162602895360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7786911 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4617887 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59213848227840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9513195 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1333030477824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60162602895360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17321720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7786911 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 951885685760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948781059072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3443007 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59213821836288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59213821836288 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50643369338368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8060894 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 950291660122 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 _step‚33 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3224210432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1875 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51976399816192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50643341203968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884284513000593e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60162602895360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7786911 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4617887 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59213848227840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9513195 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1333030477824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60162602895360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17321720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7786911 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 951885685760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948781059072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1335906725376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7804195 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50755780202496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58398573078528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17338480 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7804201 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3451037 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60299340328960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59345342316032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52091658793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9534285 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3450498 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 % _timestamp‚1.7884284521930597e+09 _runtime‚ 430.548753336 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1335962493440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7804195 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59345342316032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9555919 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚34 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4628143 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58398547650560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59347844642406 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8079180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59345315924480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 953998012928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17338480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59345315924480 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7804201 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8078638 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 950893372928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52091742695936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 952400453430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17360120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 953998012928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60296209297408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 trainer/global_step‚449óÖ¦á&ÐÛM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59347844642406 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8079180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59345315924480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 953998012928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17338480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59345315924480 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7804201 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8078638 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 950893372928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52091742695936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 952400453430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17360120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 953998012928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60296209297408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 trainer/global_step‚449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1335906725376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7804195 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50755780202496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58398573078528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17338480 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7804201 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3451037 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3932160000 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60299340328960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59345342316032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52091658793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9534285 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3450498 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 % _timestamp‚1.7884284521930597e+09 _runtime‚ 430.548753336 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1335962493440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7804195 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59345342316032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9555919 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚34 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4628143 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58398547650560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584"#éæ0Ñ+ _wandbruntime‚430  _runtime‚430E,\SÒÍ /loss/FRt4/assignment_loss‚7.211441426050095 0loss/FBt2/assignment_loss‚0.2246132737232579 .loss/FBt4/detection_loss‚6.541691414391001 1loss/FBt3/assignment_loss‚0.44377433562370544 _step‚35 1loss/SRqqt3/detection_loss‚3.5942629679389624 /loss/FRt1/detection_loss‚3.4575807397132334 .loss/FRt4/detection_loss‚2.353006986753335 1loss/SRqqt4/detection_loss‚1.7753899182146415 _runtime‚ 430.549858382 /loss/FRt3/detection_loss‚3.1734601600899124 1loss/SRqqt1/detection_loss‚3.6287322872899437 /loss/FBt2/detection_loss‚6.1375755397502365 /loss/FRt1/assignment_loss‚7.391726493835449 2loss/SRqqt2/assignment_loss‚3.2844246864318847 2loss/SRqqt3/assignment_loss‚1.9055727281324242 0loss/FBt4/assignment_loss‚0.5613439600541218 epoch‚0 trainer/global_step‚449 2loss/SRqqt4/assignment_loss‚2.4832710599899293 .loss/FRt2/detection_loss‚4.225975949655879 %loss/total_loss‚97.51500351018734 0loss/FRt2/assignment_loss‚7.2320357235995205 /loss/FRt3/assignment_loss‚7.256442032608331 1loss/SRqqt1/assignment_loss‚3.193076457732763 1loss/FBt1/assignment_loss‚0.37939995213548006 /loss/SRqqt2/detection_loss‚5.33994251092275 /loss/FBt1/detection_loss‚2.9313865781749184 .loss/FBt3/detection_loss‚12.78887632737557 $ _timestamp‚1.788428452194886e+09wldWÓÑ 1loss/SRqqt1/detection_loss‚3.6287322872899437 /loss/FBt2/detection_loss‚6.1375755397502365 /loss/FRt1/assignment_loss‚7.391726493835449 2loss/SRqqt2/assignment_loss‚3.2844246864318847 2loss/SRqqt3/assignment_loss‚1.9055727281324242 0loss/FBt4/assignment_loss‚0.5613439600541218 epoch‚0 trainer/global_step‚449 2loss/SRqqt4/assignment_loss‚2.4832710599899293 .loss/FRt2/detection_loss‚4.225975949655879 %loss/total_loss‚97.51500351018734 0loss/FRt2/assignment_loss‚7.2320357235995205 /loss/FRt3/assignment_loss‚7.256442032608331 1loss/SRqqt1/assignment_loss‚3.193076457732763 1loss/FBt1/assignment_loss‚0.37939995213548006 /loss/SRqqt2/detection_loss‚5.33994251092275 /loss/FBt1/detection_loss‚2.9313865781749184 .loss/FBt3/detection_loss‚12.78887632737557 $ _timestamp‚1.788428452194886e+09 /loss/FRt4/assignment_loss‚7.211441426050095 0loss/FBt2/assignment_loss‚0.2246132737232579 .loss/FBt4/detection_loss‚6.541691414391001 1loss/FBt3/assignment_loss‚0.44377433562370544 _step‚35 1loss/SRqqt3/detection_loss‚3.5942629679389624 /loss/FRt1/detection_loss‚3.4575807397132334 .loss/FRt4/detection_loss‚2.353006986753335 1loss/SRqqt4/detection_loss‚1.7753899182146415 _runtime‚ 430.549858382 /loss/FRt3/detection_loss‚3.1734601600899124#RèHâ‘Ô‚(:† ¨åÔç®Ëµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÈW@Õ‚(:µ ¨åÔÕ‚Ï· disk.drbd1716.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 866.47265625 disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2953.19921875$ disk./.usageGB‚265.3660430908203 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚1503.72607421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚4.078125" disk.dm-0.out‚1503.72607421875 disk.sda.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚15.78125 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.306640625 network.sent‚599742 network.recv‚242755 disk.sda.out‚15.78125 cpu‚25.105624630939452 disk.sda3.out‚15.78125 disk.drbd2080.out‚0$ memory_percent‚94.71147060394287* proc.memory.percent‚18.024897575378418 proc.cpu.threads‚17* disk./.usagePercent‚17.808937808765567 disk.sda3.in‚0 disk.nvme0n1.in‚4.078125 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚1503.72607421875.›V^žÖ‚(:“ ·åÔëÄî´ gpu.0.gpu‚78.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚46.0 gpu.0.powerWatts‚146.441' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.288200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒQ¥?ׂ(:´ ·åÔêúç¸ network.sent‚649831 disk.dm-1.out‚16.28125 disk.sda3.out‚16.28125 disk.drbd1716.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 866.44140625 disk.sda2.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0$ memory_percent‚94.71166133880615 disk.nvme0n1.in‚4.078125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2953.19921875 network.recv‚248653) disk./.usagePercent‚17.80886484661831 disk.dm-1.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚1551.56396484375 disk.drbd2080.in‚0 disk.sda1.out‚0 cpu‚25.104262107820805 disk.dm-0.in‚4.078125 disk.sda.out‚16.28125 disk.drbd2080.out‚0$ disk./.usageGB‚265.3649559020996 disk.nvme0c0n1.in‚4.078125" disk.dm-0.out‚1551.56396484375 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚1551.56396484375 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0* proc.memory.percent‚18.0248975753784183pœØ‚(:‘ ÆåԜ즵 gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚46.0 gpu.0.powerWatts‚125.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.031200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á"ôëCÙ‚(:¸ ÆåÔΖ‰· disk.nvme0c0n1.in‚4.078125 disk.sda1.in‚0 disk.drbd2266.out‚0 cpu‚25.102279253885943 disk.dm-1.in‚0 network.sent‚655971 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 16.6484375 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.078125$ disk.nvme0n1.out‚1600.6103515625# proc.memory.rssMB‚ 2953.19921875 proc.cpu.threads‚17* disk./.usagePercent‚17.811067023427842& disk.nvme0c0n1.out‚1600.6103515625 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 0.306640625 network.recv‚252945$ disk./.usageGB‚265.3977699279785 disk.sda3.out‚ 16.6484375! disk.dm-0.out‚1600.6103515625 disk.sda.out‚ 16.6484375 disk.drbd2265.out‚0( proc.memory.availableMB‚ 866.43359375* proc.memory.percent‚18.024897575378418 disk.dm-0.in‚4.078125 disk.drbd2080.in‚0 disk.rbd1.in‚0$ memory_percent‚94.71170902252197““œ¸0Ú+ _wandbruntime‚477  _runtime‚477F H¯ÃÛ½ $ _timestamp‚1.788428498793673e+09 _runtime‚ 477.148736819 _step‚36 & lr-AdamW/pg1‚8.443394778724383e-05 & lr-AdamW/pg2‚8.443394778724383e-05 trainer/global_step‚499¥ß|KÇÜÁ & lr-AdamW/pg1‚8.443394778724383e-05 & lr-AdamW/pg2‚8.443394778724383e-05 trainer/global_step‚499 $ _timestamp‚1.788428498793673e+09 _runtime‚ 477.148736819 _step‚36$饂}0Ý+ _wandbruntime‚477  _runtime‚477¤v+¤Ü'ÞÖO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5130540 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57744670068736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 _runtime‚ 477.149914642 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1052641444954 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10571047 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788428498794577e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65788552637440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8953578 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66846068429824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1480701742592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚64739000710144 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8651117 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1480757523968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1057515792384 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8651111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10592655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1054411165696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8651117 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1055744906330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66842937411584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3822492 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56263912544768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1054411165696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19243772 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10571047 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65788552637440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65788526245888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8953029 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64739026138112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65788526245888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10592655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3823038 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66842937411584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3224210432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19222158 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177¹·W„ù ßÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10592655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1054411165696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8651117 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1055744906330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66842937411584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3822492 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56263912544768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1054411165696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19243772 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10571047 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3177 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65788552637440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3932160000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65788526245888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28504489984 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8953029 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64739026138112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 [HDeviceStatsMonitor.on_train_batch_s*6*:çtart/allocated_bytes.large_pool.freed‚65788526245888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10592655 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3823038 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66842937411584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3224210432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28504489984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19222158 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3177 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5130540 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57744670068736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28504489984 _runtime‚ 477.149914642 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1657 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1052641444954 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10571047 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788428498794577e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3932160000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3177 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3177 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65788552637440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8953578 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66846068429824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3932160000 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1480701742592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚64739000710144 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8651117 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1480757523968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1057515792384 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8651111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350153728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0%“¢ '0à+ _wandbruntime‚478  _runtime‚478©FW æ&áàM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65920020334080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8668395 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5140790 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3830051 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1059628119552 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57860026492928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56376295274496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚38 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8668395 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1483649413120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1057853699638 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056523479552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19282172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65920020334080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57859944687616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1054750238054 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19282172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5140793 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10613771 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3830568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64868423139328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66976543813632 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66979674845184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10592137 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10613771 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65920046725632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8970841 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56376323408896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10592137 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64868397711360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19260532 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056523479552 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875Ñ̼ê&âäM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28504489984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10613771 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3830568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64868423139328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66976543813632 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66979674845184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10592137 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3177 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1875 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10613771 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65920046725632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3177 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8970841 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56376323408896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350153728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3932160000 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10592137 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64868397711360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19260532 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056523479552 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1875 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65920020334080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8668395 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5140790 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3830051 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1059628119552 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57860026492928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56376295274496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚38 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8668395 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1483649413120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1057853699638 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056523479552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19282172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65920020334080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28504489984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1875 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57859944687616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1054750238054 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19282172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5140793&<¿œ0ã+ _wandbruntime‚478  _runtime‚478R‹†RäÌ /loss/FRt3/assignment_loss‚7.337923332136505 %loss/total_loss‚94.40896224264678 .loss/FBt4/detection_loss‚2.253703615628183 /loss/FRt4/assignment_loss‚7.236156003815787 2loss/SRqqt2/assignment_loss‚3.2771016502380372 1loss/FBt1/assignment_loss‚0.49759203921122525 /loss/SRqqt3/detection_loss‚2.58702479644877 _step‚39 2loss/SRqqt3/assignment_loss‚2.4627700575062486 0loss/FBt4/assignment_loss‚0.4132194421221129 1loss/SRqqt4/detection_loss‚2.2202484785276417 .loss/FBt3/detection_loss‚10.76224478653499 trainer/global_step‚499 2loss/SRqqt1/assignment_loss‚2.9957997832979473 /loss/FRt3/detection_loss‚3.1603149935485537 _runtime‚ 478.044673094 0loss/FRt2/assignment_loss‚7.1830219218605444 2loss/SRqqt4/assignment_loss‚2.9504376128315926 0loss/FBt2/assignment_loss‚1.0803590267896652 epoch‚0 /loss/FRt1/detection_loss‚2.9357594993879212 .loss/FRt4/detection_loss‚2.480714914455478 0loss/SRqqt2/detection_loss‚3.211091437935829 /loss/FBt1/detection_loss‚3.6064392382446306 /loss/FBt2/detection_loss‚11.454910636485325 /loss/FRt1/assignment_loss‚7.384818562921488 .loss/FRt2/detection_loss‚3.486121472559477 0loss/SRqqt1/detection_loss‚2.781090580194262 0loss/FBt3/assignment_loss‚0.6500983599645613 % _timestamp‚1.7884284996897418e+09ÎùW@VåÐ 0loss/FBt3/assignment_loss‚0.6500983599645613 % _timestamp‚1.7884284996897418e+09 /loss/FRt3/assignment_loss‚7.337923332136505 %loss/total_loss‚94.40896224264678 .loss/FBt4/detection_loss‚2.253703615628183 /loss/FRt4/assignment_loss‚7.236156003815787 2loss/SRqqt2/assignment_loss‚3.2771016502380372 1loss/FBt1/assignment_loss‚0.49759203921122525 /loss/SRqqt3/detection_loss‚2.58702479644877 _step‚39 2loss/SRqqt3/assignment_loss‚2.4627700575062486 0loss/FBt4/assignment_loss‚0.4132194421221129 1loss/SRqqt4/detection_loss‚2.2202484785276417 .loss/FBt3/detection_loss‚10.76224478653499 trainer/global_step‚499 2loss/SRqqt1/assignment_loss‚2.9957997832979473 /loss/FRt3/detection_loss‚3.1603149935485537 _runtime‚ 478.044673094 0loss/FRt2/assignment_loss‚7.1830219218605444 2loss/SRqqt4/assignment_loss‚2.9504376128315926 0loss/FBt2/assignment_loss‚1.0803590267896652 epoch‚0 /loss/FRt1/detection_loss‚2.9357594993879212 .loss/FRt4/detection_loss‚2.480714914455478 0loss/SRqqt2/detection_loss‚3.211091437935829 /loss/FBt1/detection_loss‚3.6064392382446306 /loss/FBt2/detection_loss‚11.454910636485325 /loss/FRt1/assignment_loss‚7.384818562921488 .loss/FRt2/detection_loss‚3.486121472559477 0loss/SRqqt1/detection_loss‚2.781090580194262'rv_‘æ‚(:† ÕåÔãàе gpu.0.gpu‚60.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82087707519531* gpu.0.memoryAllocatedBytes‚ 29051912192 gpu.0.temp‚47.0 gpu.0.powerWatts‚134.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.898 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç?Hç‚(:½ ÕåÔ¥¢“¸ disk.sda3.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚662698 network.recv‚258177 disk.dm-1.out‚ 17.19921875 disk.nvme0c0n1.in‚4.078125 disk.sda3.out‚ 17.19921875 disk.drbd1716.in‚0( proc.memory.availableMB‚ 866.34765625 cpu‚25.100123468879083 disk.dm-0.in‚4.078125" disk.dm-0.out‚1641.41259765625 disk.sda1.in‚0% disk./.usageGB‚265.39826583862305 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.306640625# proc.memory.rssMB‚ 2953.19921875% disk.nvme0n1.out‚1641.41259765625' disk.nvme0c0n1.out‚1641.41259765625 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda.out‚ 17.19921875 disk.sda2.in‚0* proc.memory.percent‚18.024897575378418 disk.drbd2266.out‚0 disk.rbd1.in‚0" memory_percent‚94.712233543396 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚17.811100304407297 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.078125 disk.drbd2266.in‚0B‚Ek“è‚(:ˆ äåÔëªÔµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82331848144531* gpu.0.memoryAllocatedBytes‚ 29054009344 gpu.0.temp‚48.0 gpu.0.powerWatts‚266.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ÿ“Jé‚(:¿ äåÔòþ›· disk.rbd1.out‚ 0.306640625 disk.drbd2080.in‚0 disk.sda.in‚0* proc.memory.percent‚18.024897575378418 network.recv‚263347% disk.nvme0n1.out‚1696.92431640625( proc.memory.availableMB‚ 866.33984375# proc.memory.rssMB‚ 2953.19921875 disk.drbd2266.in‚0 disk.sda3.out‚ 17.62890625 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0% disk./.usageGB‚265.42505645751953 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚1696.92431640625 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚17.812898245320262$ memory_percent‚94.71228122711182 cpu‚25.098772046470334 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 17.62890625 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚712218 disk.dm-1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-1.out‚ 17.62890625 disk.rbd1.in‚0 disk.dm-0.in‚4.078125" disk.dm-0.out‚1696.92431640625 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125ßÈÅ7ê‚(:ˆ óåÔ—µñ´ gpu.0.gpu‚93.0 gpu.0.µºß\memory‚17+ gpu.0.memoryAllocated‚33.82331848144531* gpu.0.memoryAllocatedBytes‚ 29054009344 gpu.0.temp‚49.0 gpu.0.powerWatts‚299.624' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©±´u@ë‚(:µ óåÔüÔ÷ cpu‚25.097844307352695* disk./.usagePercent‚17.815329548872942" disk.dm-0.out‚1742.55322265625 disk.rbd0.in‚0 disk.dm-1.out‚ 18.328125 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚265.4612846374512 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda.out‚ 18.328125 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚1742.55322265625& proc.memory.availableMB‚ 866.078125# proc.memory.rssMB‚ 2953.19921875 disk.sda3.out‚ 18.328125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚718594 network.recv‚267968 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚1742.55322265625 disk.sda2.in‚0# memory_percent‚94.7138786315918 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0* proc.memory.percent‚18.024897575378418 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.306640625SO/…0ì+ _wandbruntime‚524  _runtime‚524цÒ1Âí¼ _step‚40 % lr-AdamW/pg1‚9.28942631967873e-05 % lr-AdamW/pg2‚9.28942631967873e-05 trainer/global_step‚549 % _timestamp‚1.7884285462827547e+09 _runtime‚ 524.637835186Ñ1¬ÆîÀ _runtime‚ 524.637835186 _step‚40 % lr-AdamW/pg1‚9.28942631967873e-05 % lr-AdamW/pg2‚9.28942631967873e-05 trainer/global_step‚549 % _timestamp‚1.7884285462827547e+09(üR¤>0ï+ _wandbruntime‚524  _runtime‚524Puk›Ý'ð×O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73526402946048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72363230655488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73526402946048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9849997 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4206807 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3178 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3934257152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73523271927808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28506587136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9515317 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72363257047040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73523271927808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4206267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 trainer/global_step‚549 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72363257047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71208850770944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21165824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350178304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72363230655488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1158094691162 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1163145899008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1628158240768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1628104556544 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1876 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21144210 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9515317 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61884427616768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3178 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11650507 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63512532173312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28506587136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11650507 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1160041272320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73526402946048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72363230655488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73526402946048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9849997 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4206807 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3178 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3934257152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73523271927808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28506587136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9515317 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72363257047040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73523271927808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4206267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 trainer/global_step‚549 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72363257047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71208850770944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21165824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 350178304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72363230655488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1158094691162 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1163145899008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1628158240768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1628104556544 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1876 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21144210 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9515317 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61884427616768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3178 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11650507 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63512532173312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28506587136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11650507 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1160041272320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0)ÇÄÖ#“ò‚(:ˆ ‚‚åÔÙ劵 gpu.0.gpu‚85.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82331848144531* gpu.0.memoryAllocatedBytes‚ 29054009344 gpu.0.temp‚45.0 gpu.0.powerWatts‚148.947' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.7894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸tæ=ó‚(:² ‚‚åÔ¤Ëù¶# proc.memory.rssMB‚ 2953.19921875 disk.drbd1716.out‚0 disk.sda.in‚0 disk.sda.out‚ 18.6796875 disk.nvme0n1.in‚4.078125# disk.nvme0n1.out‚1811.283203125 disk.dm-0.out‚1811.283203125% proc.memory.availableMB‚ 866.28125 cpu‚25.096596102063554 disk.sda3.out‚ 18.6796875 disk.drbd1716.in‚0 disk.drbd2080.in‚0* disk./.usagePercent‚17.818312804669446 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.078125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.306640625% disk.nvme0c0n1.out‚1811.283203125 disk.dm-1.out‚ 18.6796875* proc.memory.percent‚18.024897575378418 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0 network.sent‚724705 network.recv‚272039$ disk./.usageGB‚265.5057373046875 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚94.71263885498047"Åù0ô+ _wandbruntime‚525  _runtime‚525˜ýã­j"õáM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71338247772160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3934257152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3934257152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72494751135232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9532595 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9532601 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11649989 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1657 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63627893419008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28506587136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 525.554502836 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21204224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21204224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5653443 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1163306945846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3178 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3178 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21182584 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63627975224320 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72498451256422 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1160203484262 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28506587136 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867415 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28506587136 % _timestamp‚1.7884285471989338e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11671623 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3934257152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1876 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72501580145974 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11671623 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11649989 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350178304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5653440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61996838480896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1631108609024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9532601 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 trainer/global_step‚549 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4214519 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73656878329856 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9532595 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61996866615296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1162153586176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73660009361408 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21204224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5653443 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1163306945846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3178 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3178 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21182584 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63627975224320 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72498451256422 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1160203484262 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28506587136 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867415 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28506587136 % _timestamp‚1.7884285471989338e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11671623 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3934257152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1876 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72501580145974 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11671623 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11649989 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 350178304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5653440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61996838480896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1631108609024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9532601 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 trainer/global_step‚549 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4214519 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73656878329856 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9532595 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3224210432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61996866615296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1162153586176*NB0÷+ _wandbruntime‚525  _runtime‚525a³#KøÅ .loss/FBt2/detection_loss‚5.820672988891602 .loss/FBt3/detection_loss‚4.861254789705934 -loss/FRt1/detection_loss‚4.61937289165728 /loss/FRt2/assignment_loss‚6.976720968882243 0loss/FRt3/assignment_loss‚7.3057441626276285 /loss/FRt4/assignment_loss‚7.185171604156494 .loss/FRt3/detection_loss‚2.642180909369407 0loss/FRt1/assignment_loss‚7.3954674836361045 2loss/SRqqt2/assignment_loss‚2.9637590921842136 0loss/FBt3/assignment_loss‚0.7109880905602416 0loss/FBt4/assignment_loss‚0.4614355552308115 /loss/FBt4/detection_loss‚2.4664746306414873 $loss/total_loss‚86.3439131144493 .loss/FRt2/detection_loss‚3.748901685078939 2loss/SRqqt4/assignment_loss‚2.7672604395793035 0loss/SRqqt3/detection_loss‚3.301416875834161 0loss/SRqqt4/detection_loss‚1.379174296227892 /loss/FRt4/detection_loss‚2.7618085441162004 0loss/SRqqt1/detection_loss‚3.591243935128053 trainer/global_step‚549 .loss/FBt1/detection_loss‚2.660806698639204 epoch‚0 $ _timestamp‚1.788428547200606e+09 1loss/SRqqt1/assignment_loss‚3.164336701234182 1loss/SRqqt3/assignment_loss‚2.095184744672572 0loss/FBt1/assignment_loss‚1.0813731293270135 _step‚43 _runtime‚ 525.555583576 0loss/FBt2/assignment_loss‚0.3381985824150099 0loss/SRqqt2/detection_loss‚6.0449643146533238×\`OùÉ /loss/FRt4/detection_loss‚2.7618085441162004 0loss/SRqqt1/detection_loss‚3.591243935128053 trainer/global_step‚549 .loss/FBt1/detection_loss‚2.660806698639204 epoch‚0 $ _timestamp‚1.788428547200606e+09 1loss/SRqqt1/assignment_loss‚3.164336701234182 1loss/SRqqt3/assignment_loss‚2.095184744672572 0loss/FBt1/assignment_loss‚1.0813731293270135 _step‚43 _runtime‚ 525.555583576 0loss/FBt2/assignment_loss‚0.3381985824150099 0loss/SRqqt2/detection_loss‚6.044964314653323 .loss/FBt2/detection_loss‚5.820672988891602 .loss/FBt3/detection_loss‚4.861254789705934 -loss/FRt1/detection_loss‚4.61937289165728 /loss/FRt2/assignment_loss‚6.976720968882243 0loss/FRt3/assignment_loss‚7.3057441626276285 /loss/FRt4/assignment_loss‚7.185171604156494 .loss/FRt3/detection_loss‚2.642180909369407 0loss/FRt1/assignment_loss‚7.3954674836361045 2loss/SRqqt2/assignment_loss‚2.9637590921842136 0loss/FBt3/assignment_loss‚0.7109880905602416 0loss/FBt4/assignment_loss‚0.4614355552308115 /loss/FBt4/detection_loss‚2.4664746306414873 $loss/total_loss‚86.3439131144493 .loss/FRt2/detection_loss‚3.748901685078939 2loss/SRqqt4/assignment_loss‚2.7672604395793035 0loss/SRqqt3/detection_loss‚3.301416875834161 0loss/SRqqt4/detection_loss‚1.379174296227892+†Ð¦=žú‚(:“ ‘‚åԢꪵ gpu.0.gpu‚53.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82331848144531* gpu.0.memoryAllocatedBytes‚ 29054009344 gpu.0.temp‚48.0 gpu.0.powerWatts‚287.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.493399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SËš¿Aû‚(:¶ ‘‚åÔåÊÜ· disk.dm-0.in‚4.078125 disk.sda.in‚0 cpu‚25.095289934342773 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 0.306640625 disk.dm-0.out‚1852.861328125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0$ memory_percent‚94.71299648284912 disk.sda3.in‚0 disk.sda3.out‚ 19.26171875 disk.drbd2080.in‚0) disk./.usagePercent‚17.81986216226667# disk.nvme0n1.out‚1852.861328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.078125 disk.rbd1.in‚0 disk.sda.out‚ 19.26171875 disk.rbd0.out‚0( proc.memory.availableMB‚ 866.22265625! proc.memory.rssMB‚ 2953.203125 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚18.024921417236328 network.sent‚763718 network.recv‚277982% disk./.usageGB‚265.52882385253906 disk.dm-1.out‚ 19.26171875 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚1852.861328125^÷gï‘ü‚(:†  ‚åÔŸ‹ú´ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82331848144531* gpu.0.memoryAllocatedBytes‚ 29054009344 gpu.0.temp‚45.0 gpu.0.powerWatts‚198.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r%&=ý‚(:²  ‚åÔ¨™½· disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚94.71306800842285 cpu‚25.0937110329272 network.sent‚802350$ disk./.usageGB‚265.5474281311035 disk.nvme0n1.in‚4.078125 disk.rbd1.out‚ 0.306640625 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚4.078125 network.recv‚282306 disk.drbd2265.out‚0$ disk.nvme0n1.out‚1893.5595703125 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.drbd2266.out‚0! disk.dm-0.out‚1893.5595703125 disk.sda3.in‚0! proc.memory.rssMB‚ 2953.203125 proc.cpu.threads‚17 disk.sda.out‚ 19.6640625 disk.sda2.in‚0 disk.sda3.out‚ 19.6640625 disk.sda1.out‚0) disk./.usagePercent‚17.82111071101116 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 19.6640625 disk.nvme0c0n1.in‚4.078125 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚1893.5595703125' proc.memory.availableMB‚ 866.2109375* proc.memory.percent‚18.024921417236328aX¤É“þ‚(:ˆ ¯‚åÔ‚›µ gpu.0.gpu‚76.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚46.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÆNþAÿ‚(:¶ ¯‚åÔ£îζ disk.rbd0.out‚0 disk.dm-1.out‚ 20.5390625 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚17 disk.drbd2080.out‚0$ memory_percent‚94.71466541290283) proc.memory.percent‚18.02494525909424# disk./.usageGB‚265.575496673584& disk.nvme0c0n1.out‚1930.2900390625! disk.dm-0.out‚1930.2900390625 disk.sda.out‚ 20.5390625( proc.memory.availableMB‚ 865.94921875 network.sent‚808702 disk.rbd0.in‚0 disk.nvme0n1.in‚4.078125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2953.20703125 cpu‚25.092685473997726 disk.nvme0c0n1.in‚4.078125 disk.sda3.out‚ 20.5390625 disk.drbd2265.in‚0$ disk.nvme0n1.out‚1930.2900390625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚286974* disk./.usagePercent‚17.822994414448097 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.dm-0.in‚4.078125 disk.drbd2080.in‚0–‹F0€+ _wandbruntime‚572  _runtime‚572êìEÆÀ ' lr-AdamW/pg1‚0.00010135457860633077 ' lr-AdamW/pg2‚0.00010135457860633077 trainer/global_step‚599 % _timestamp‚1.7884285937415202e+09 _runtime‚ 572.096559777 _step‚44?o7OÊ‚Ä trainer/global_step‚599 % _timestamp‚1.7884285937415202e+09 _runtime‚ 572.096559777 _step‚44 ' lr-AdamW/pg1‚0.00010135457860633077 ' lr-AdamW/pg2‚0.00010135457860633077,9–‰ý0ƒ+ _wandbruntime‚572  _runtime‚572YšÒKâ'„ÜO XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78945377658458 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1265671378944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1775378145280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 _runtime‚ 572.097658932 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10379517 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23066262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12686751 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77678700831744 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80203606444032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78937961456640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚88110080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78937935065088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10379517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80206737462272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6155837 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78942248769114 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1265671378944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12708359 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10741769 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4585932 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 371596288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1877 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1263547937370 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69280348968448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69280348968448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocationz}Öâs.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 _runtime‚ 572.097658932 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10379517 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23066262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12686751 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77678700831744 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80203606444032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78937961456640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚88110080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78937935065088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10379517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80206737462272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6155837 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78942248769114 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1265671378944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12708359 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10741769 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4585932 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 371596288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1877 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1263547937370 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6166090 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10396795 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚561 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79076883452982 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3936354304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1778299992064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 _step‚46 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79069429153280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10759537 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1678 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1778357857280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1270888332800 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85999616 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79069429153280 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6166093 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57865216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23126276 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23126276 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12707841 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77808123260928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80340343877632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79069455544832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80340343877632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77808097832960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1268760192054 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12707841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79069455544832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4594008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67617381687296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3179 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23104636 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1267783692800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3243084800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12729475 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949í~Oñ&ˆëM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1778299992064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 _step‚46 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79069429153280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10759537 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1678 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1778357857280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1270888332800 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85999616 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79069429153280 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6166093 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57865216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23126276 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23126276 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12707841 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77808123260928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80340343877632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79069455544832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80340343877632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77808097832960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1268760192054 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12707841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79069455544832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4594008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67617381687296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3179 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23104636 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1267783692800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3243084800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12729475 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6166090 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10396795 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚561 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79076883452982 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3936354304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3.—g´r0‰+ _wandbruntime‚573  _runtime‚573f…ƤOŠÉ 0loss/FBt3/assignment_loss‚0.5525381648395149 1loss/SRqqt4/detection_loss‚2.2961419153943217 /loss/FBt2/detection_loss‚2.1303123533725743 _step‚47 -loss/FBt3/detection_loss‚8.01801419784041 /loss/FBt4/detection_loss‚2.1776242777705193 epoch‚0 _runtime‚ 573.010010564 2loss/SRqqt1/assignment_loss‚2.6139008051728547 .loss/FRt2/detection_loss‚3.487432211637497 /loss/FRt4/assignment_loss‚7.156532120286372 trainer/global_step‚599 1loss/SRqqt4/assignment_loss‚2.717506678104401 0loss/FBt4/assignment_loss‚0.4070601573712947 % _timestamp‚1.7884285946550398e+09 /loss/FRt1/assignment_loss‚7.411228593190511 /loss/FRt3/assignment_loss‚7.209339866638183 0loss/FBt2/assignment_loss‚0.5365504051247948 0loss/SRqqt1/detection_loss‚1.636183997637302 %loss/total_loss‚84.02544170738881 2loss/SRqqt2/assignment_loss‚3.2487251281738283 .loss/FRt3/detection_loss‚2.959340156391263 /loss/FBt1/detection_loss‚6.3254419228059255 0loss/FRt2/assignment_loss‚7.1067193150520325 /loss/FRt1/detection_loss‚3.1656306319766574 -loss/FRt4/detection_loss‚2.39018128586835 0loss/SRqqt2/detection_loss‚4.440914735623768 0loss/SRqqt3/detection_loss‚2.772067383715981 2loss/SRqqt3/assignment_loss‚2.4455133320992455 0loss/FBt1/assignment_loss‚0.8205420713012035ˆëÆS‹Í /loss/FRt1/detection_loss‚3.1656306319766574 -loss/FRt4/detection_loss‚2.39018128586835 0loss/SRqqt2/detection_loss‚4.440914735623768 0loss/SRqqt3/detection_loss‚2.772067383715981 2loss/SRqqt3/assignment_loss‚2.4455133320992455 0loss/FBt1/assignment_loss‚0.8205420713012035 0loss/FBt3/assignment_loss‚0.5525381648395149 1loss/SRqqt4/detection_loss‚2.2961419153943217 /loss/FBt2/detection_loss‚2.1303123533725743 _step‚47 -loss/FBt3/detection_loss‚8.01801419784041 /loss/FBt4/detection_loss‚2.1776242777705193 epoch‚0 _runtime‚ 573.010010564 2loss/SRqqt1/assignment_loss‚2.6139008051728547 .loss/FRt2/detection_loss‚3.487432211637497 /loss/FRt4/assignment_loss‚7.156532120286372 trainer/global_step‚599 1loss/SRqqt4/assignment_loss‚2.717506678104401 0loss/FBt4/assignment_loss‚0.4070601573712947 % _timestamp‚1.7884285946550398e+09 /loss/FRt1/assignment_loss‚7.411228593190511 /loss/FRt3/assignment_loss‚7.209339866638183 0loss/FBt2/assignment_loss‚0.5365504051247948 0loss/SRqqt1/detection_loss‚1.636183997637302 %loss/total_loss‚84.02544170738881 2loss/SRqqt2/assignment_loss‚3.2487251281738283 .loss/FRt3/detection_loss‚2.959340156391263 /loss/FBt1/detection_loss‚6.3254419228059255 0loss/FRt2/assignment_loss‚7.1067193150520325/ZÛ“Œ‚(:ˆ ¾‚åÔÈý—µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°HÚ¨H‚(:½ ¾‚åԦɷ disk.sda3.out‚ 20.87890625% disk./.usageGB‚265.58494567871094 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1979.78564453125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0$ memory_percent‚94.71688270568848 network.sent‚815019 disk.sda.in‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚1979.78564453125 disk.dm-1.out‚ 20.87890625" disk.dm-0.out‚1979.78564453125* disk./.usagePercent‚17.823628545110413 disk.sda.out‚ 20.87890625 disk.rbd0.in‚0 disk.nvme0n1.in‚4.078125 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.306640625 disk.dm-0.in‚4.078125 network.recv‚291752 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2953.20703125 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0) proc.memory.percent‚18.02494525909424 cpu‚25.091146411967525' proc.memory.availableMB‚ 865.5859375žg_>žŽ‚(:“ Í‚åÔ˜–µ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚48.0 gpu.0.powerWatts‚287.662' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.532399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L+^qG‚(:¼ Í‚åÔÙßñ¶ disk.drbd2265.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2953.20703125 cpu‚25.090216702903334* disk./.usagePercent‚17.824664351593945 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 network.recv‚298457 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚4.078125% disk.nvme0n1.out‚2018.69873046875 disk.dm-1.in‚0$ memory_percent‚94.71540451049805 disk.sda3.out‚ 21.58203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 21.58203125 disk.sda2.out‚0 network.sent‚865507% disk./.usageGB‚265.60037994384766 disk.drbd2080.in‚0& proc.memory.availableMB‚ 865.828125' disk.nvme0c0n1.out‚2018.69873046875 disk.sda2.in‚0 disk.dm-1.out‚ 21.58203125 disk.dm-0.in‚4.078125" disk.dm-0.out‚2018.69873046875 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0) proc.memory.percent‚18.02494525909424Ám 9*‚(:’ Ü‚åÔ”¡ºµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0Â?Çs.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m–m›F‘‚(:» Ü‚åÔ¹†˜· disk.nvme0c0n1.in‚4.078125' disk.nvme0c0n1.out‚2049.01806640625 network.sent‚871391 disk.sda3.out‚ 22.17578125 disk.drbd2265.in‚0 disk.sda3.in‚0 cpu‚25.08964131427666 disk.sda.out‚ 22.17578125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚4.078125$ disk./.usageGB‚265.6139831542969 disk.drbd2080.out‚0 disk.dm-1.out‚ 22.17578125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625) proc.memory.percent‚18.02494525909424 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.recv‚302937 disk.rbd0.out‚0% disk.nvme0n1.out‚2049.01806640625 disk.dm-0.in‚4.078125$ memory_percent‚94.71385478973389# proc.memory.rssMB‚ 2953.20703125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚2049.01806640625( proc.memory.availableMB‚ 866.08203125) disk./.usagePercent‚17.82557727446105 disk.drbd1716.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0pÑ#0’+ _wandbruntime‚619  _runtime‚619UVÆ“À ' lr-AdamW/pg2‚0.00010981489401587423 trainer/global_step‚649 % _timestamp‚1.7884286408584056e+09 _runtime‚ 619.213586509 _step‚48 ' lr-AdamW/pg1‚0.00010981489401587423ÄZtÊ”Ä ' lr-AdamW/pg2‚0.00010981489401587423 trainer/global_step‚649 % _timestamp‚1.7884286408584056e+09 _runtime‚ 619.213586509 _step‚48 ' lr-AdamW/pg1‚0.000109814894015874230~FÏæ0•+ _wandbruntime‚619  _runtime‚619ò¹g¯ã'–ÝO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25009928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13766211 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1877 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3179 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4966282 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1922446927872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369001183578 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1371301485568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13744603 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1371301485568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73125514029568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚86012928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85512639474688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75047989091840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1372104644954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11635337 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3936354304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 trainer/global_step‚649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6668487 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86887071978496 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86883940960256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73125542163968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1877 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4966847 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11243711 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84148550892544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11243717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85520680965466 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86883940960256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75047903078912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84148576320512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86883940960256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75047903078912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84148576320512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11635337 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3936354304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 trainer/global_step‚6491 /0˜+ _wandbruntime‚620  _runtime‚620y!Åí&™çM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84277947893760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87020678393856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11261001 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1373413799424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25048328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73237953028096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6678743 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1925381611008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3179 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11652622 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85652186759990 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1374213438262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884286417451737e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13787327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1373413799424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371109976678 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚50 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85644159954432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84277973321728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85644133562880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1925325842944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87017547362304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55768064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25026688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11653164 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85644159954432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11260995 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75163334639104 _runtime‚ 620.100924075 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73237924893696¡p^L#šëM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1373413799424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371109976678 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚50 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85644159954432 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84277973321728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85644133562880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1925325842944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87017547362304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55768064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25026688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11653164 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85644159954432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11260995 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75163334639104 _runtime‚ 620.100924075 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73237924893696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84277947893760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87020678393856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11261001 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1373413799424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25048328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73237953028096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6678743 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1925381611008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3179 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11652622 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85652186759990 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1374213438262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884286417451737e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚137873272»™”0›+ _wandbruntime‚620  _runtime‚620–s“PQœË trainer/global_step‚649 .loss/FRt3/assignment_loss‚7.31380012387135 /loss/FRt4/detection_loss‚2.4707956295765814 1loss/SRqqt3/detection_loss‚3.1383024958219936 /loss/FRt1/assignment_loss‚7.370060159609868 2loss/SRqqt2/assignment_loss‚3.2973557608468194 1loss/SRqqt1/detection_loss‚2.7089880349859596 1loss/SRqqt4/detection_loss‚1.1550436942627476 /loss/FBt2/detection_loss‚10.582520448244601 _step‚51 /loss/FBt4/detection_loss‚2.3933170389522007 /loss/FRt2/assignment_loss‚7.007949612357399 0loss/FBt2/assignment_loss‚0.8169025113770308 0loss/SRqqt2/detection_loss‚4.338148159640176 .loss/FBt1/detection_loss‚8.101415494501037 .loss/FBt3/detection_loss‚6.067843294956467 % _timestamp‚1.7884286417472706e+09 .loss/FRt1/detection_loss‚2.868497419815797 2loss/SRqqt4/assignment_loss‚2.5728083153565726 /loss/FRt3/detection_loss‚2.5728721437884157 epoch‚0 0loss/FBt1/assignment_loss‚0.6023604906433775 /loss/FRt2/detection_loss‚3.3181153644214976 %loss/total_loss‚91.31984559569739 _runtime‚ 620.102221625 1loss/FBt4/assignment_loss‚0.27657965857268346 /loss/FRt4/assignment_loss‚7.331324022324359 1loss/SRqqt1/assignment_loss‚2.471523891091347 1loss/SRqqt3/assignment_loss‚2.066385642011115 /loss/FBt3/assignment_loss‚0.476936188668002ÒŒ‘”UÏ /loss/FBt4/detection_loss‚2.3933170389522007 /loss/FRt2/assignment_loss‚7.007949612357399 0loss/FBt2/assignment_loss‚0.8169025113770308 0loss/SRqqt2/detection_loss‚4.338148159640176 .loss/FBt1/detection_loss‚8.101415494501037 .loss/FBt3/detection_loss‚6.067843294956467 % _timestamp‚1.7884286417472706e+09 .loss/FRt1/detection_loss‚2.868497419815797 2loss/SRqqt4/assignment_loss‚2.5728083153565726 /loss/FRt3/detection_loss‚2.5728721437884157 epoch‚0 0loss/FBt1/assignment_loss‚0.6023604906433775 /loss/FRt2/detection_loss‚3.3181153644214976 %loss/total_loss‚91.31984559569739 _runtime‚ 620.102221625 1loss/FBt4/assignment_loss‚0.27657965857268346 /loss/FRt4/assignment_loss‚7.331324022324359 1loss/SRqqt1/assignment_loss‚2.471523891091347 1loss/SRqqt3/assignment_loss‚2.066385642011115 /loss/FBt3/assignment_loss‚0.476936188668002 trainer/global_step‚649 .loss/FRt3/assignment_loss‚7.31380012387135 /loss/FRt4/detection_loss‚2.4707956295765814 1loss/SRqqt3/detection_loss‚3.1383024958219936 /loss/FRt1/assignment_loss‚7.370060159609868 2loss/SRqqt2/assignment_loss‚3.2973557608468194 1loss/SRqqt1/detection_loss‚2.7089880349859596 1loss/SRqqt4/detection_loss‚1.1550436942627476 /loss/FBt2/detection_loss‚10.582520448244601 _step‚513NqÉ“ž‚(:ˆ ë‚åÔÿ唵 gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚48.0 gpu.0.powerWatts‚291.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å 9HŸ‚(:½ ë‚åÔ¶Âá·# proc.memory.rssMB‚ 2953.20703125) proc.memory.percent‚18.02494525909424 disk.sda3.out‚ 22.73828125 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.sda2.in‚0' disk.nvme0c0n1.out‚2084.39697265625 disk.sda1.out‚0 cpu‚25.088308845745264" disk.dm-0.out‚2084.39697265625 disk.drbd2080.in‚0% disk.nvme0n1.out‚2084.39697265625 disk.rbd1.out‚ 0.306640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.826162251676788$ memory_percent‚94.71895694732666 network.recv‚306752 disk.drbd2265.out‚0 disk.dm-1.out‚ 22.73828125 network.sent‚876561 disk.sda.out‚ 22.73828125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 865.24609375 disk.dm-0.in‚4.078125 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚265.6226997375488 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚4.078125 ðSá ‚(:’ ú‚åÔŸÕø´ gpu.0.gpu‚74.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.092000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*‰A¡‚(:¶ ú‚åÔûĺ·$ disk./.usageGB‚265.6422576904297 disk.dm-1.out‚ 23.26953125 network.recv‚312124 disk.sda.out‚ 23.26953125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2953.20703125 disk.rbd1.out‚ 0.306640625 disk.sda3.out‚ 23.26953125 disk.drbd1716.out‚0) proc.memory.percent‚18.02494525909424 disk.sda.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 865.73828125 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.078125# disk.nvme0n1.out‚2126.865234375 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-0.out‚2126.865234375 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚94.71595287322998 disk.rbd0.in‚0% disk.nvme0c0n1.out‚2126.865234375 disk.sda1.out‚0 disk.drbd2080.out‚0 cpu‚25.08710233873783 network.sent‚926373* disk./.usagePercent‚17.827474802304838 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.078125P>D “¢‚(:ˆ ‰ƒåÔÄöø´ gpu.0.gpu‚92.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­f A£‚(:¶ ‰ƒåԧз disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0# disk.nvme0n1.out‚2170.919921875 disk.drbd2266.out‚0 cpu‚25.086277662200228$ disk./.usageGB‚265.6497802734375 disk.drbd1716.in‚0 disk.sda.out‚ 23.91796875$ memory_percent‚94.71595287322998( proc.memory.availableMB‚ 865.73828125) proc.memory.percent‚18.02494525909424 disk.dm-1.out‚ 23.91796875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.306640625# proc.memory.rssMB‚ 2953.20703125 network.recv‚316733 disk.dm-0.in‚4.078125 disk.sda3.out‚ 23.91796875 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚2170.919921875 network.sent‚932687 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚17.82797964916236 disk.dm-0.out‚2170.919921875 disk.rbd0.out‚0 disk.drbd2266.in‚0/ ý‘0¤+ _wandbruntime‚666  _runtime‚666&‡ ÂÆ¥À % _timestamp‚1.7884286882508137e+09 _runtime‚ 666.605840919 _step‚52 ' lr-AdamW/pg1‚0.00011827520942541771 ' lr-AdamW/pg2‚0.00011827520942541771 trainer/global_step‚699"â±Ê¦Ä trainer/global_step‚699 % _timestamp‚1.7884286882508137e+09 _runtime‚ 666.605840919 _step‚52 ' lr-AdamW/pg1‚0.00011827520942541771 ' lr-AdamW/pg2‚0.000118275209425417714€v*0§+ _wandbruntime‚666  _runtime‚666·Ã¼Âã'¨ÝO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1480036218880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93564275476480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1480036218880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1474454429786 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28508684288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12107917 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90618400953344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26910366 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5346142 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92087370275840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92087343884288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7181140 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1476931592192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2069369075712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14802455 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚935674064947208ˆŒ©áO trainer/global_step‚699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90618400953344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26910366 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5346142 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92087370275840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92087343884288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7181140 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1476931592192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2069369075712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14802455 QIDeviceStatsMonito$Úg…[r.on_train_batch_start/inactive_split.small_pool.current‚564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93567406494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1480036218880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93564275476480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1480036218880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1474454429786 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28508684288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12107917 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚05T×U`0ª+ _wandbruntime‚667  _runtime‚667Ù±ôì&«æM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12125195 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14845179 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92224361177446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3179 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1482148546048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7191390 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5353655 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80930799191552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92227490066998 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14823545 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5353138 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2072331091456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93697881878528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92218837972480 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3243084800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1482148546048 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93701012910080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78858496234496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7191393 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26970380 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26948740 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12125201 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14823545 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92218864364032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1678 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80930883094016 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14845179 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1479043906048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90747797954560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26970380 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1479043906048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12544528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚54 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93697881878528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92218837972480 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3243084800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1482148546048 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93701012910080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78858496234496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7191393 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26970380 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26948740 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12125201 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14823545 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92218864364032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1678 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80930883094016 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14845179 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1479043906048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90747797954560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26970380 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1479043906048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12544528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚54 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86583034655232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98662074685440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1585666325504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84366600442368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12972111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28832418 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12972111 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84366628576768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1579907675994 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15881915 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98671287579482 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28508684288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13417614 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100247741010944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57878528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98662048293888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7693790 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1582561698816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2216406078464 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98668158690138 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100244609992704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1582561698816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7693787 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5723263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 375197696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1877 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100244609992704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1877 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1583011137370 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100247741010944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98662074685440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97088276442112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1585666325504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13417050 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86582948642304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15881915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15860307 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5723824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97088251014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15860307 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚86012928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 _runtime‚ 714.006305514 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86583034655232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98662074685440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1585666325504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84366600442368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12972111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28832418 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12972111 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84366628576768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1579907675994 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15881915 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98671287579482 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28508684288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13417614 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100247741010944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57878528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98662048293888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7693790 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1582561698816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2216406078464 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98668158690138 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100244609992704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1582561698816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7693787 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5723263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 375197696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1877 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100244609992704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1877 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1583011137370 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100247741010944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98662074685440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97088276442112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1585666325504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13417050 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86582948642304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15881915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15860307 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5723824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97088251014144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15860307 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚86012928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 _runtime‚ 714.006305514 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13435340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28892432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28870792 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15903031 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 _runtime‚ 714.901197667 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12989395 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1585119930678 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100381347426304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1587778652672 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86698295506432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3249376256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97217648015360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28508684288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚88096768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28892432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5731297 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84479039440896 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98802793374006 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 _step‚58 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1584674012672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15881397 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86698383603200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97217673443328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98793542382080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚554 % _timestamp‚1.7884287365450099e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12989395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 375197696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28508684288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98793568773632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100381347426304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2219344162304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2219284199936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1587778652672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7704040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3179 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5730743 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1582016469094 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13434783 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3936354304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15881397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1678 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98793542382080 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28892432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5731297 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84479039440896 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98802793374006 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 _step‚58 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1584674012672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15881397 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86698383603200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97217673443328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98793542382080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚554 % _timestamp‚1.7884287365450099e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12989395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 375197696 >8DeviceStatsMonitor.on_train_batch_eððEAnd/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28508684288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98793568773632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100381347426304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2219344162304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2219284199936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1587778652672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7704040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3179 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5730743 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1582016469094 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13434783 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3936354304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15881397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1678 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98793542382080 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13435340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28892432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28870792 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15903031 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 _runtime‚ 714.901197667 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12989395 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1585119930678 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100381347426304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1587778652672 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86698295506432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3249376256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97217648015360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28508684288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚88096768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0:PÞ\0¿+ _wandbruntime‚714  _runtime‚714¼’¡¯LÀÆ 1loss/SRqqt4/detection_loss‚1.5909552087680674 epoch‚0 /loss/FRt4/assignment_loss‚7.287740707397461 2loss/SRqqt3/assignment_loss‚2.5469003457289476 1loss/FBt4/assignment_loss‚0.28792929393927225 /loss/FBt1/detection_loss‚1.6896277786309466 _step‚59 1loss/SRqqt1/detection_loss‚2.1666028091913803 -loss/FBt2/detection_loss‚7.19749837120374 /loss/FRt3/assignment_loss‚7.311131251942028 2loss/SRqqt4/assignment_loss‚2.5116542622224607 0loss/SRqqt2/detection_loss‚5.078192369214126 /loss/FBt4/detection_loss‚1.7581839749473147 _runtime‚ 714.902611005 0loss/FBt3/assignment_loss‚0.5174511392628226 .loss/FRt1/detection_loss‚3.187120762540073 %loss/total_loss‚81.79604517897047 trainer/global_step‚749 0loss/FBt2/assignment_loss‚0.4644774707655112 /loss/FRt1/assignment_loss‚7.482543933682326 -loss/FRt2/detection_loss‚2.67673649285969 /loss/FRt4/detection_loss‚2.1802527609012894 .loss/FBt3/detection_loss‚6.313025020501193 -loss/FRt2/assignment_loss‚7.1063064274035 2loss/SRqqt2/assignment_loss‚2.7586777210235596 0loss/SRqqt3/detection_loss‚3.734514525303474 % _timestamp‚1.7884287365476046e+09 0loss/SRqqt1/assignment_loss‚2.83354501397002 0loss/FBt1/assignment_loss‚0.5362533337029163 .loss/FRt3/detection_loss‚2.578724203868346ñ’«PÁÊ 1loss/SRqqt1/detection_loss‚2.1666028091913803 -loss/FBt2/detection_loss‚7.19749837120374 /loss/FRt3/assignment_loss‚7.311131251942028 2loss/SRqqt4/assignment_loss‚2.5116542622224607 0loss/SRqqt2/detection_loss‚5.078192369214126 /loss/FBt4/detection_loss‚1.7581839749473147 _runtime‚ 714.902611005 0loss/FBt3/assignment_loss‚0.5174511392628226 .loss/FRt1/detection_loss‚3.187120762540073 %loss/total_loss‚81.79604517897047 trainer/global_step‚749 0loss/FBt2/assignment_loss‚0.4644774707655112 /loss/FRt1/assignment_loss‚7.482543933682326 -loss/FRt2/detection_loss‚2.67673649285969 /loss/FRt4/detection_loss‚2.1802527609012894 .loss/FBt3/detection_loss‚6.313025020501193 -loss/FRt2/assignment_loss‚7.1063064274035 2loss/SRqqt2/assignment_loss‚2.7586777210235596 0loss/SRqqt3/detection_loss‚3.734514525303474 % _timestamp‚1.7884287365476046e+09 0loss/SRqqt1/assignment_loss‚2.83354501397002 0loss/FBt1/assignment_loss‚0.5362533337029163 .loss/FRt3/detection_loss‚2.578724203868346 1loss/SRqqt4/detection_loss‚1.5909552087680674 epoch‚0 /loss/FRt4/assignment_loss‚7.287740707397461 2loss/SRqqt3/assignment_loss‚2.5469003457289476 1loss/FBt4/assignment_loss‚0.28792929393927225 /loss/FBt1/detection_loss‚1.6896277786309466 _step‚59;DÖ¢•“‚(:ˆ ŃåÔ÷„—µ gpu.0.gpu‚86.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚47.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì+h9Â(:® ŃåÔþ–¸ disk.sda3.out‚ 26.453125 disk.drbd1716.out‚0 disk.sda2.in‚0 network.sent‚ 1.000169e+06 disk.sda3.in‚0 disk.sda.out‚ 26.453125 disk.drbd2265.in‚0! disk.nvme0n1.out‚ 2361.4609375 disk.dm-0.in‚4.078125 disk.dm-0.out‚ 2361.4609375) proc.memory.percent‚18.02496910095215 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.455078125 cpu‚25.082985934921265 disk.sda1.out‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.out‚ 26.453125 disk.nvme0c0n1.in‚4.078125' proc.memory.availableMB‚ 865.6171875" proc.memory.rssMB‚ 2953.2109375 network.recv‚333868 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0# memory_percent‚94.7166919708252 proc.cpu.threads‚17$ disk./.usageGB‚265.6724853515625 disk.sda2.out‚0* disk./.usagePercent‚17.829503406006157 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.out‚ 2361.4609375’¶žÄ‚(:“ ÔƒåÔéüŽµ gpu.0.gpu‚47.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.839200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qg…ÑHÅ‚(:½ ÔƒåÔØù¶$ disk.nvme0n1.out‚2403.5888671875 disk.drbd2266.out‚0 disk.sda3.out‚ 26.82421875) proc.memory.percent‚18.02499294281006$ disk./.usageGB‚265.6821174621582 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0$ memory_percent‚94.71526145935059' proc.memory.availableMB‚ 865.8515625 disk.sda.out‚ 26.82421875 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125# proc.memory.rssMB‚ 2953.21484375 disk.sda2.out‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.out‚ 26.82421875 disk.rbd1.in‚0 disk.rbd1.out‚ 0.455078125 proc.cpu.threads‚17* disk./.usagePercent‚17.830149825030116 disk.dm-0.in‚4.078125 disk.sda1.in‚0 disk.drbd2080.out‚0 cpu‚25.0824675964296 disk.rbd0.out‚0& disk.nvme0c0n1.out‚2403.5888671875 disk.sda3.in‚0 network.sent‚ 1.049224e+06 network.recv‚338545 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0! disk.dm-0.out‚2403.5888671875 disk.drbd2080.in‚0Ì$ÓÆ‚(:ˆ ãƒåÔÞ䎵 gpu.0.gpu‚55.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚49.0 gpu.0.powerWatts‚299.516' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CM5Þ8Ç‚(:­ ãƒåÔãÅí¶ network.sent‚ 1.055186e+06 disk.nvme0n1.in‚4.078125 disk.rbd1.out‚ 0.4873046875' disk.nvme0c0n1.out‚2441.35595703125 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚27.5 disk.drbd2266.out‚0 disk.sda3.out‚27.5) proc.memory.percent‚18.02499294281006 cpu‚25.081797833323836 disk.sda2.in‚0" disk.dm-0.out‚2441.35595703125 disk.sda1.in‚0# memory_percent‚94.7152853012085( proc.memory.availableMB‚ 865.84765625# proc.memory.rssMB‚ 2953.21484375 disk.drbd2266.in‚0 disk.dm-0.in‚4.078125 disk.sda.out‚27.5* disk./.usagePercent‚17.826631769494586 disk.dm-1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.sda.in‚0 network.recv‚342992# disk./.usageGB‚265.629695892334% disk.nvme0n1.out‚2441.35595703125 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0¡À8>0È+ _wandbruntime‚760  _runtime‚7600d©ÆÉÀ ' lr-AdamW/pg2‚0.00013519584024450465 trainer/global_step‚799 % _timestamp‚1.7884287825624793e+09 _runtime‚ 760.917540613 _step‚60 ' lr-AdamW/pg1‚0.00013519584024450465 ^š%ÊÊÄ % _timestamp‚1.7884287825624793e+09 _runtime‚ 760.917540613 _step‚60 ' lr-AdamW/pg1‚0.00013519584024450465 ' lr-AdamW/pg2‚0.00013519584024450465 trainer/global_step‚799<ݳ…0Ë+ _wandbruntime‚760  _runtime‚760Mhqï'ÌéO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106924944508928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106928075527168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6104147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89987143648768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13836311 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚ 760.918936147 _step‚61 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚86012928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30776084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105236779095040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691296432128 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92349938489856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1688464383578 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106924944508928 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92350024502784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3261959168 trainer/global_step‚799 % _timestamp‚1.7884287825634534e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105243461997146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14310033 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3179 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6103596 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16939767 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105236752703488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103558101074944 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13836317 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16918159 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28508684288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3936354304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30754470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1691296432128 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106924944508928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106928075527168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6104147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89987143648768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13836311 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚ 760.918936147 _step‚61 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚86012928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30776084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105236779095040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691296432128 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92349938489856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1688464383578 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106924944508928 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92350024502784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3261959168 trainer/global_step‚799 % _timestamp‚1.7884287825634534e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105243461997146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14310033 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3179 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6103596 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16939767 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105236752703488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103558101074944 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13836317 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16918159 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28508684288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3936354304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30754470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1691296432128 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90099554512896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2365727721984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105378096681014 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13853595 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14327773 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85999616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30814484 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13853595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3179 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105374967791462 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105368273183232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1687469715302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107058550910976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1693408759296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103687523504128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92465282234880 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107061681942528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 387910656 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16939249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105368273183232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 trainer/global_step‚799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105368246791680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13853601 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3179 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16960883 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 _step‚62 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90099582647296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103687498076160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30792844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30814484 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1693408759296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107061681942528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8216693 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1690304119296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 761.816479499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6111627ŠäÕ'ý&Ð÷M PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30814484 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1693408759296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107061681942528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8216693 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1690304119296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3179 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 761.816479499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6111627 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92465368234496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28508684288 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30792844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6111083 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3261959168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1690304119296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90099554512896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2365727721984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105378096681014 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13853595 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14327773 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85999616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30814484 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13853595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3179 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚949 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105374967791462 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105368273183232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1687469715302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107058550910976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1693408759296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103687523504128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92465282234880 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107061681942528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 387910656 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16939249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105368273183232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 trainer/global_step‚799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105368246791680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13853601 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3179 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16960883 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 _step‚62 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90099582647296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103687498076160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3179 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30792844>µ…«+0Ñ+ _wandbruntime‚761  _runtime‚761‡ÏÙ3OÒÉ /loss/FRt2/assignment_loss‚6.916948117707905 2loss/SRqqt2/assignment_loss‚3.1195202532567476 0loss/FBt1/assignment_loss‚0.5007924142604073 /loss/FRt2/detection_loss‚2.0777275813253304 /loss/FRt4/detection_loss‚2.2740815417574987 0loss/SRqqt3/detection_loss‚2.565329827900444 $loss/total_loss‚96.1601672520747 2loss/SRqqt3/assignment_loss‚2.3091244543237344 % _timestamp‚1.7884287834626148e+09 1loss/SRqqt1/assignment_loss‚2.206017764210701 0loss/FBt2/assignment_loss‚0.5719486670568585 .loss/FRt1/detection_loss‚2.945232544404765 /loss/FRt3/detection_loss‚2.2291352888118596 .loss/SRqqt2/detection_loss‚3.6416088138756 epoch‚0 trainer/global_step‚799 2loss/SRqqt4/assignment_loss‚2.6144083876540694 1loss/FBt3/assignment_loss‚0.23035123043357292 0loss/FBt4/assignment_loss‚0.2615184362210632 .loss/FBt1/detection_loss‚7.132782702023785 _runtime‚ 761.817608029 0loss/SRqqt1/detection_loss‚2.332503983307397 .loss/FBt3/detection_loss‚10.89808393186993 /loss/FBt4/detection_loss‚1.7739313409183524 0loss/FRt4/assignment_loss‚7.2382509878703525 /loss/FRt1/assignment_loss‚7.149258583784103 /loss/FRt3/assignment_loss‚7.361696520159321 1loss/SRqqt4/detection_loss‚1.5218316422694402 .loss/FBt2/detection_loss‚16.28808223667147 _step‚63±ÔžSÓÍ 1loss/SRqqt1/assignment_loss‚2.206017764210701 0loss/FBt2/assignment_loss‚0.5719486670568585 .loss/FRt1/detection_loss‚2.945232544404765 /loss/FRt3/detection_loss‚2.2291352888118596 .loss/SRqqt2/detection_loss‚3.6416088138756 epoch‚0 trainer/global_step‚799 2loss/SRqqt4/assignment_loss‚2.6144083876540694 1loss/FBt3/assignment_loss‚0.23035123043357292 0loss/FBt4/assignment_loss‚0.2615184362210632 .loss/FBt1/detection_loss‚7.132782702023785 _runtime‚ 761.817608029 0loss/SRqqt1/detection_loss‚2.332503983307397 .loss/FBt3/detection_loss‚10.89808393186993 /loss/FBt4/detection_loss‚1.7739313409183524 0loss/FRt4/assignment_loss‚7.2382509878703525 /loss/FRt1/assignment_loss‚7.149258583784103 /loss/FRt3/assignment_loss‚7.361696520159321 1loss/SRqqt4/detection_loss‚1.5218316422694402 .loss/FBt2/detection_loss‚16.28808223667147 _step‚63 /loss/FRt2/assignment_loss‚6.916948117707905 2loss/SRqqt2/assignment_loss‚3.1195202532567476 0loss/FBt1/assignment_loss‚0.5007924142604073 /loss/FRt2/detection_loss‚2.0777275813253304 /loss/FRt4/detection_loss‚2.2740815417574987 0loss/SRqqt3/detection_loss‚2.565329827900444 $loss/total_loss‚96.1601672520747 2loss/SRqqt3/assignment_loss‚2.3091244543237344 % _timestamp‚1.7884287834626148e+09?9·ÀÔ‚(:’ òƒåÔ•Ô‡µ gpu.0.gpu‚89.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚48.0 gpu.0.powerWatts‚298.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.748000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|µWBJÕ‚(:¿ òƒåÔהݷ disk.drbd2080.out‚0 disk.sda2.in‚0" disk.dm-0.out‚2497.00244140625 disk.sda3.out‚ 28.02734375 proc.memory.rssMB‚ 2953.21875 network.sent‚ 1.060972e+06 disk.drbd2265.in‚0% disk.nvme0n1.out‚2497.00244140625 disk.drbd2266.in‚0) proc.memory.percent‚18.02501678466797 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.4873046875 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125) disk./.usagePercent‚17.82804672313633 disk.sda.out‚ 28.02734375 disk.nvme0n1.in‚4.078125' disk.nvme0c0n1.out‚2497.00244140625 disk.sda3.in‚0 disk.sda1.in‚0 cpu‚25.080822164245635 proc.cpu.threads‚17$ disk./.usageGB‚265.6507797241211 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 28.02734375 disk.drbd2266.out‚0 network.recv‚347199 disk.drbd1716.in‚0 disk.rbd1.in‚0$ memory_percent‚94.71583366394043' proc.memory.availableMB‚ 865.7578125 disk.drbd1716.out‚0s «™žÖ‚(:“ „åÔÌÈŠµ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚46.0 gpu.0.powerWatts‚204.395' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.879000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤œcÏKׂ(:À „åÔ’' proc.memory.availableMB‚ 865.5078125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.sent‚ 1.111145e+06 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.55029296875 disk.dm-0.in‚4.078125$ memory_percent‚94.71735954284668 network.recv‚353085) disk./.usagePercent‚17.82674185273431 disk.dm-1.out‚ 28.68359375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚25.080651125179394 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125' disk.nvme0c0n1.out‚2532.73291015625) proc.memory.percent‚18.02501678466797 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.sda2.out‚0" disk.dm-0.out‚2532.73291015625 proc.memory.rssMB‚ 2953.21875$ disk./.usageGB‚265.6313362121582 disk.sda3.out‚ 28.68359375 disk.drbd1716.out‚0 disk.sda.out‚ 28.68359375% disk.nvme0n1.out‚2532.73291015625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125 +uØ‚(:’ „åÔÎûóµ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82575988769531* gpu.0.memoryAllocatedBytes‚ 29056106496 gpu.0.temp‚48.0 gpu.0.powerWatts‚261.613' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.322599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´±ð»DÙ‚(:¹ „åÔõšþ¸ disk.drbd2265.out‚0 disk.rbd1.in‚0# memory_percent‚92.9534912109375 proc.memory.rssMB‚ 2953.21875) proc.memory.percent‚18.02501678466797 cpu‚25.079919102008056 disk.sda.out‚ 29.15234375 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125' disk.nvme0c0n1.out‚2596.75634765625 disk.dm-0.in‚4.078125% disk.nvme0n1.out‚2596.75634765625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0) disk./.usagePercent‚17.82858357093563 disk.sda.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 network.sent‚ 1.11639e+06 disk.rbd1.out‚ 0.55029296875 disk.sda3.in‚0 disk.sda3.out‚ 29.15234375" proc.memory.availableMB‚1154.5 disk.nvme0n1.in‚4.078125 disk.sda2.out‚0 disk.dm-1.out‚ 29.15234375 disk.sda1.in‚0" disk.dm-0.out‚2596.75634765625 network.recv‚356667$ disk./.usageGB‚265.6587791442871 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0‰¥_0Ú+ _wandbruntime‚808  _runtime‚808s’ÈŸÄÛ¾ trainer/global_step‚849 # _timestamp‚1.78842883019047e+09 _runtime‚ 808.545502135 _step‚64 ' lr-AdamW/pg1‚0.00014365615565404812 ' lr-AdamW/pg2‚0.00014365615565404812û·gÒÈÜ _step‚64 ' lr-AdamW/pg1‚0.00014365615565404812 ' lr-AdamW/pg2‚0.00014365615565404812 trainer/global_step‚849 # _timestamp‚1.78842883019047e+09 _runtime‚ 808.545502135@ó.»š0Ý+ _wandbruntime‚808  _runtime‚808Hká!ÞéO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14700511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3179 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111811457113088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17976011 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1678 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1790814168410 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28508684288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113605279025152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6483666 =7DeviceStatsMonitor.on_train_batÉ’´XÎ!ch_start/num_device_free‚0 _step‚65 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14700511 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2510316658688 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1793821912064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111818765304154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110027976563712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98117949829632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95607714989568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111821894193498 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6484189 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95607686855168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1796926538752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1793821912064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14700517 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 808.546805816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3936354304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1796926538752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3261959168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 387910656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1877 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111811457113088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2510262974464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17976011 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15203279 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113608410043392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111811483504640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 % _timestamp‚1.7884288301913772e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8719087 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1793917629786 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32698136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3179 trainer/global_step‚849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113605279025152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111811483504640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15202753 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98118031648256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17997619 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8719090 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2510316658688 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1793821912064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111818765304154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3936354304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110027976563712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3179 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3179 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98117949829632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95607714989568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111821894193498 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6484189 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95607686855168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28508684288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1796926538752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1793821912064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14700517 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 808.546805816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3936354304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1796926538752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3261959168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 387910656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1877 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111811457113088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2510262974464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17976011 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15203279 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113608410043392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111811483504640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 % _timestamp‚1.7884288301913772e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8719087 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1793917629786 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3936354304 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32698136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3179 trainer/global_step‚849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113605279025152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111811483504640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15202753Aö³0à+ _wandbruntime‚809  _runtime‚809kßH.ù&áóM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1795934225920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8729340 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98233385176064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113742016458752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111953399988022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32736536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8729343 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18018735 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15220670 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110157373564928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1799038865920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111942977592832 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98233301273600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32736536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95720125853696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2513203554304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32714896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 _step‚66 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111942977592832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32714896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14717801 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111942951201280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 387910656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17997101 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113742016458752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3936354304 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 % _timestamp‚1.7884288310897644e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14717795 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6491839 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18018735 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111942951201280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1877 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1796026423094 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28508684288‹^ óâ÷M trainer/global_step‚849 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15221182 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110157348136960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14717795 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1795934225920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15220670 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110157373564928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1799038865920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111942977592832 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98233301273600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32736536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95720125853696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2513203554304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32714896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 _step‚66 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111942977592832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32714896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14717801 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111942951201280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1877 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 387910656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3936354304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17997101 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113742016458752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3936354304 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 % _timestamp‚1.7884288310897644e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14717795 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6491839 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18018735 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111942951201280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_bat¦ðÇ ch_end/segment.small_pool.peak‚1877 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1796026423094 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28508684288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1795934225920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8729340 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98233385176064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28508684288 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1877 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113742016458752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111953399988022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32736536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8729343 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18018735 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3936354304 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59975680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1899452018688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118386187914240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6865493 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1896267414618 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚ 855.911169839 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19033863 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120285613541376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16097811 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101228230061568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 % _timestamp‚1.7884288775559127e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19055471 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1899370875994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚578 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1902556645376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15564711 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2656958044160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6866071 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116497801196544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚899 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1902556645376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34620188 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚581 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15564717 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚88110080 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59975680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1899452018688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118386187914240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6865493 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1896267414618 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚ 855.911169839 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19033863 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120285613541376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16097811 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101228230061568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 % _timestamp‚1.7884288775559127e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19055471 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1899370875994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚578 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1902556645376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15564711 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2656958044160EÝ?›i0ô+ _wandbruntime‚856  _runtime‚856†¢Byü&õöM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120419219943424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101340669060096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 $ _timestamp‚1.788428878454976e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116627198197760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1901479669302 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19054953 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118517655610880 _step‚70 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1904668972544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15581995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15582001 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118517682002432 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6873586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚118525574405478 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120422350974976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34658588 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1904668972544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104000458512896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15581995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2659877549568 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34658588 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16115017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1901564332544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34636948 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19076587 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120422350974976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚899 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104000546609664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9241990 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¦$·'öúM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15582001 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118517682002432 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6873586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚118525574405478 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120422350974976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34658588 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1904668972544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104000458512896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15581995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2659877549568 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34658588 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16115017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1901564332544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34636948 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19076587 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120422350974976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚899 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104000546609664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9241990 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120419219943424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101340669060096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 $ _timestamp‚1.788428878454976e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116627198197760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1901479669302 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19054953 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118517655610880 _step‚70 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1904668972544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15581995Fr"Ò0÷+ _wandbruntime‚856  _runtime‚856w¦»ÕRøÌ .loss/FRt1/detection_loss‚2.595537090053161 /loss/FRt2/detection_loss‚3.0752980375289916 1loss/SRqqt3/detection_loss‚1.7010464435836798 .loss/FBt4/detection_loss‚1.174527265855383 epoch‚0 trainer/global_step‚899 2loss/SRqqt2/assignment_loss‚2.8945141047239304 /loss/FRt1/assignment_loss‚5.763867417971293 1loss/SRqqt1/detection_loss‚2.4707619154706912 0loss/SRqqt2/detection_loss‚3.153139704465866 .loss/FBt3/detection_loss‚6.651587679982185 1loss/FBt2/assignment_loss‚0.33983138608627167 0loss/FBt3/assignment_loss‚0.6708392103992082 /loss/FBt1/assignment_loss‚0.643086176414631 1loss/FBt4/assignment_loss‚0.49513839467552306 /loss/FRt4/detection_loss‚2.8838820051639638 1loss/SRqqt4/detection_loss‚2.6339314634702644 /loss/FBt2/detection_loss‚3.3774874318729746 /loss/FRt2/assignment_loss‚5.714289646148682 /loss/FRt3/assignment_loss‚5.397002580030909 /loss/FRt4/assignment_loss‚6.829645136569408 1loss/SRqqt1/assignment_loss‚2.517869792086013 2loss/SRqqt4/assignment_loss‚2.0659344439603844 $loss/total_loss‚70.9302372133637 _runtime‚ 856.812542109 _step‚71 % _timestamp‚1.7884288784574943e+09 2loss/SRqqt3/assignment_loss‚1.9742953869658457 /loss/FRt3/detection_loss‚2.5156636226851985 .loss/FBt1/detection_loss‚3.391060877199236/Þj1VùÐ /loss/FRt1/assignment_loss‚5.763867417971293 1loss/SRqqt1/detection_loss‚2.4707619154706912 0loss/SRqqt2/detection_loss‚3.153139704465866 .loss/FBt3/detection_loss‚6.651587679982185 1loss/FBt2/assignment_loss‚0.33983138608627167 0loss/FBt3/assignment_loss‚0.6708392103992082 /loss/FBt1/assignment_loss‚0.643086176414631 1loss/FBt4/assignment_loss‚0.49513839467552306 /loss/FRt4/detection_loss‚2.8838820051639638 1loss/SRqqt4/detection_loss‚2.6339314634702644 /loss/FBt2/detection_loss‚3.3774874318729746 /loss/FRt2/assignment_loss‚5.714289646148682 /loss/FRt3/assignment_loss‚5.397002580030909 /loss/FRt4/assignment_loss‚6.829645136569408 1loss/SRqqt1/assignment_loss‚2.517869792086013 2loss/SRqqt4/assignment_loss‚2.0659344439603844 $loss/total_loss‚70.9302372133637 _runtime‚ 856.812542109 _step‚71 % _timestamp‚1.7884288784574943e+09 2loss/SRqqt3/assignment_loss‚1.9742953869658457 /loss/FRt3/detection_loss‚2.5156636226851985 .loss/FBt1/detection_loss‚3.391060877199236 .loss/FRt1/detection_loss‚2.595537090053161 /loss/FRt2/detection_loss‚3.0752980375289916 1loss/SRqqt3/detection_loss‚1.7010464435836798 .loss/FBt4/detection_loss‚1.174527265855383 epoch‚0 trainer/global_step‚899 2loss/SRqqt2/assignment_loss‚2.8945141047239304GÎ"žú‚(:“ Û„åԃȵ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.066' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.413199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02sÊFû‚(:» Û„åÔ­ƒø¶ disk.drbd1716.in‚0 disk.sda2.out‚0 cpu‚25.076974883289385* disk./.usagePercent‚17.835508830744367% disk./.usageGB‚265.76197052001953 disk.rbd1.out‚ 0.61328125 disk.drbd1716.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0 network.recv‚379130 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda.out‚ 31.9296875$ disk.nvme0n1.out‚2794.0849609375 disk.dm-1.out‚ 31.9296875) proc.memory.availableMB‚ 1180.06640625 proc.memory.rssMB‚ 2953.21875 disk.sda1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚4.078125 disk.sda2.in‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125! disk.dm-0.out‚2794.0849609375 disk.sda3.out‚ 31.9296875 disk.drbd2080.out‚0$ memory_percent‚92.79744625091553) proc.memory.percent‚18.02501678466797& disk.nvme0c0n1.out‚2794.0849609375 disk.rbd0.in‚0 network.sent‚ 1.190008e+06ý‚7cžü‚(:“ ê„åÔæ³Ùµ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.839200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦¨ïáIý‚(:¾ ê„åÔÂã· disk.rbd1.out‚ 0.61328125) disk./.usagePercent‚17.83656511783064$ disk./.usageGB‚265.7777099609375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0 network.recv‚384271' disk.nvme0c0n1.out‚2835.25341796875 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚2835.25341796875 disk.sda2.out‚0) proc.memory.availableMB‚ 1179.77734375 network.sent‚ 1.239717e+06 disk.dm-0.in‚4.078125 disk.sda3.in‚0 disk.sda3.out‚ 32.40234375 disk.sda.out‚ 32.40234375) proc.memory.percent‚18.02501678466797 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 32.40234375$ memory_percent‚92.79921054840088 proc.memory.rssMB‚ 2953.21875 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚2835.25341796875 disk.nvme0n1.in‚4.078125 cpu‚25.07624638548336˜{Xžþ‚(:“ ù„åÔ®íªµ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚134.865' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.973000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖcøzBÿ‚(:· ù„åÔ®©Ù¶ proc.memory.rssMB‚ 2953.21875 disk.rbd1.in‚0 disk.sda3.out‚ 33.015625 disk.drbd2080.in‚0) proc.memory.percent‚18.02501678466797 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚4.078125" disk.dm-0.out‚2870.10107421875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1180.0234375* disk./.usagePercent‚17.837112461938847 disk.rbd1.out‚ 0.61328125 disk.dm-0.in‚4.078125 disk.sda1.out‚0 disk.dm-1.in‚0$ disk./.usageGB‚265.7858657836914' disk.nvme0c0n1.out‚2870.10107421875 disk.drbd2080.out‚0 disk.sda.out‚ 33.015625 disk.drbd2266.in‚0 network.recv‚388847 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 cpu‚25.07584445753188 network.sent‚ 1.24598e+06 disk.sda1.in‚0 disk.dm-1.out‚ 33.015625$ memory_percent‚92.79770851135254 proc.cpu.threads‚17% disk.nvme0n1.out‚2870.101074218756d: 0€+ _wandbruntime‚903  _runtime‚903¤ú®+ÆÀ ' lr-AdamW/pg2‚0.00016057678647313505 trainer/global_step‚949 % _timestamp‚1.7884289249330964e+09 _runtime‚ 903.288117632 _step‚72 ' lr-AdamW/pg1‚0.00016057678647313505–RßNÊ‚Ä ' lr-AdamW/pg1‚0.00016057678647313505 ' lr-AdamW/pg2‚0.00016057678647313505 trainer/global_step‚949 % _timestamp‚1.7884289249330964e+09 _runtime‚ 903.288117632 _step‚72H™y±²0ƒ+ _wandbruntime‚903  _runtime‚903|<²,%„íO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126969079075840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124960892323840 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2004824122202 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16428911 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2005082125312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2803932367872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124960865932288 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚73 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20113323 PCDeviceStatsMonitor.oE&™ZÎ!n_train_batch_start/allocation.large_pool.freed‚16428911 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16989163 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 % _timestamp‚1.7884289249338303e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36520626 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20113323 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20091715 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2005082125312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16428917 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7244240 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2803876586496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126965948057600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7244773 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9744387 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16428917 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124960892323840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124960865932288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124972500807514 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106848773267968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124969371918170 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 32724449288½CE÷'…ñO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚73 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20113323 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16428911 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16989163 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 % _timestamp‚1.7884289249338303e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36520626 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20113323 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20091715 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2005082125312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16428917 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7244240 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2803876586496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126965948057600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7244773 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9744387 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16428917 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124960892323840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124960865932288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124972500807514 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106848773267968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124969371918170 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126969079075840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124960892323840 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2004824122202 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16428911 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2005082125312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2803932367872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124960865932288 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608IÁGq0†+ _wandbruntime‚904  _runtime‚904ÎÑ Gý&‡÷M >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17006534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125104006602038 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127099554459648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125092386412032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36580640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9754643 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2010299079168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2007194439168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2003829453926 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125100877712486 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36580640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20134439 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36559000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20134439 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚ 904.175112634 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106961184132096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚74 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16446195 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7252418 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109767996664832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2806868300800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36559000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2010299079168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16446195 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16446201 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125092360020480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123097048258560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20112805 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17007061 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106961212266496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2006932915510 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 % _timestamp‚1.7884289258195987e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127102685491200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7251894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123097073686528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125092386412032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109768080567296 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864mÔëˆûM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36580640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20134439 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36559000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20134439 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚ 904.175112634 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106961184132096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚74 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16446195 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7252418 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109767996664832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2806868300800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36559000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2010299079168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16446195 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16446201 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125092360020480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123097048258560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.currentÞZr‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20112805 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17007061 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106961212266496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2006932915510 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 % _timestamp‚1.7884289258195987e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127102685491200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7251894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123097073686528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125092386412032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109768080567296 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17006534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125104006602038 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127099554459648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125092386412032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36580640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9754643 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2010299079168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2007194439168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2003829453926 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125100877712486 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878J&3I0‰+ _wandbruntime‚904  _runtime‚904ÇýA“VŠÐ 0loss/FBt4/assignment_loss‚0.2633222802411316 /loss/FRt3/assignment_loss‚4.547899423761571 1loss/SRqqt2/assignment_loss‚2.804278059703548 1loss/SRqqt4/assignment_loss‚2.364932960585544 0loss/FBt1/assignment_loss‚0.6066229827385189 1loss/SRqqt4/detection_loss‚3.0122306903726175 /loss/FBt4/detection_loss‚1.8894670098776767 /loss/FRt4/assignment_loss‚7.178858105341593 .loss/FBt3/detection_loss‚5.653477327691184 /loss/FRt2/detection_loss‚2.8470810695930764 .loss/FRt4/detection_loss‚3.646616515517235 1loss/SRqqt3/detection_loss‚1.6820616995294888 _runtime‚ 904.176143049 0loss/FRt1/assignment_loss‚4.8691220689327155 1loss/SRqqt1/detection_loss‚1.8762318983412625 1loss/SRqqt2/detection_loss‚3.0102796278348785 % _timestamp‚1.7884289258211737e+09 /loss/FRt1/detection_loss‚2.5300663646231305 1loss/FBt2/assignment_loss‚0.23287341481773183 1loss/FBt3/assignment_loss‚0.42637424039559035 /loss/FBt1/detection_loss‚2.7364772076812547 1loss/SRqqt3/assignment_loss‚1.697572424243658 .loss/FRt3/detection_loss‚2.760467460814943 %loss/total_loss‚68.45629688403184 epoch‚0 _step‚75 0loss/FRt2/assignment_loss‚5.3395969602796765 2loss/SRqqt1/assignment_loss‚2.2266863579289957 /loss/FBt2/detection_loss‚4.2537007331848145 trainer/global_step‚949–r¿2Z‹Ô /loss/FRt2/detection_loss‚2.8470810695930764 .loss/FRt4/detection_loss‚3.646616515517235 1loss/SRqqt3/detection_loss‚1.6820616995294888 _runtime‚ 904.176143049 0loss/FRt1/assignment_loss‚4.8691220689327155 1loss/SRqqt1/detection_loss‚1.8762318983412625 1loss/SRqqt2/detection_loss‚3.0102796278348785 % _timestamp‚1.7884289258211737e+09 /loss/FRt1/detection_loss‚2.5300663646231305 1loss/FBt2/assignment_loss‚0.23287341481773183 1loss/FBt3/assignment_loss‚0.42637424039559035 /loss/FBt1/detection_loss‚2.7364772076812547 1loss/SRqqt3/assignment_loss‚1.697572424243658 .loss/FRt3/detection_loss‚2.760467460814943 %loss/total_loss‚68.45629688403184 epoch‚0 _step‚75 0loss/FRt2/assignment_loss‚5.3395969602796765 2loss/SRqqt1/assignment_loss‚2.2266863579289957 /loss/FBt2/detection_loss‚4.2537007331848145 trainer/global_step‚949 0loss/FBt4/assignment_loss‚0.2633222802411316 /loss/FRt3/assignment_loss‚4.547899423761571 1loss/SRqqt2/assignment_loss‚2.804278059703548 1loss/SRqqt4/assignment_loss‚2.364932960585544 0loss/FBt1/assignment_loss‚0.6066229827385189 1loss/SRqqt4/detection_loss‚3.0122306903726175 /loss/FBt4/detection_loss‚1.8894670098776767 /loss/FRt4/assignment_loss‚7.178858105341593 .loss/FBt3/detection_loss‚5.653477327691184K”ÔžŒ‚(:“ ˆ…åԚجµ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚185.191' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.038199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íc™=‚(:² ˆ…åÔ‘¹‰· disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.out‚33.4375 disk.sda3.in‚0 proc.memory.rssMB‚ 2953.21875 network.sent‚ 1.251261e+06 disk.drbd2080.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 0.61328125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.recv‚392668 disk.drbd2266.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚2910.96630859375% disk.nvme0n1.out‚2910.96630859375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚2910.96630859375 cpu‚25.075411369880854) disk./.usagePercent‚17.83756661930459 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.out‚33.4375 disk.drbd2265.in‚0) proc.memory.percent‚18.02501678466797$ disk./.usageGB‚265.7926330566406 disk.dm-0.in‚4.078125 disk.sda3.out‚33.4375 disk.rbd0.in‚0# memory_percent‚92.7997350692749 disk.sda2.out‚0 disk.nvme0c0n1.in‚4.078125) proc.memory.availableMB‚ 1179.69140625 proc.cpu.threads‚17”ˆ @“Ž‚(:ˆ —…åÔ÷Œµ gpu.0.gpu‚69.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚268.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G6¾J‚(:¿ —…åÔ¸ïÚ· disk.dm-1.out‚ 34.3046875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.out‚ 34.3046875 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚398394' disk.nvme0c0n1.out‚2947.27880859375 disk.sda2.out‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2953.22265625 cpu‚25.075240122189555$ disk./.usageGB‚265.7961311340332 disk.sda.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 34.3046875 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125% disk.nvme0n1.out‚2947.27880859375 network.sent‚ 1.301198e+06 disk.rbd1.out‚ 0.61328125" disk.dm-0.out‚2947.27880859375 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1179.19140625) proc.memory.percent‚18.02504062652588* disk./.usagePercent‚17.837801378213488 disk.dm-0.in‚4.078125 disk.rbd0.out‚0 disk.drbd2266.in‚0# memory_percent‚92.8027868270874 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0%Z—“‚(:ˆ ¦…åÔÇ«£µ gpu.0.gpu‚83.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Ò{-K‘‚(:À ¦…åÔ㱂·$ disk./.usageGB‚265.8099136352539 disk.dm-0.in‚4.078125" disk.dm-0.out‚2985.86376953125 disk.sda3.out‚ 34.76171875 disk.drbd1716.in‚0) proc.memory.percent‚18.02504062652588 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0$ memory_percent‚92.79978275299072# proc.memory.rssMB‚ 2953.22265625 proc.cpu.threads‚17 disk.rbd0.in‚0 network.recv‚402300 disk.drbd2266.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚2985.86376953125 disk.sda2.out‚0) proc.memory.availableMB‚ 1179.68359375 cpu‚25.07488676227438 disk.sda.in‚0 disk.drbd2265.in‚0 network.sent‚ 1.307113e+06 disk.rbd1.out‚ 0.61328125 disk.sda3.in‚0( disk./.usagePercent‚17.8387263334347 disk.dm-1.in‚0 disk.dm-1.out‚ 34.76171875 disk.sda1.in‚0 disk.sda.out‚ 34.76171875 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.078125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚2985.86376953125 disk.drbd1716.out‚0 (O–0’+ _wandbruntime‚950  _runtime‚950<Т0Æ“À ' lr-AdamW/pg1‚0.00016903710188267854 ' lr-AdamW/pg2‚0.00016903710188267854 trainer/global_step‚999 % _timestamp‚1.7884289722591166e+09 _runtime‚ 950.614156478 _step‚76ˆËîÊ”Ä % _timestamp‚1.7884289722591166e+09 _runtime‚ 950.614156478 _step‚76 ' lr-AdamW/pg1‚0.00016903710188267854 ' lr-AdamW/pg2‚0.00016903710188267854 trainer/global_step‚999LÚQS0•+ _wandbruntime‚950  _runtime‚950²ø.õ'–ïO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21171175 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131547804114522 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚999 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21149567 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17293117 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2110712231936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17293111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚ 950.615147268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10257037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133646282573824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131535596733440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133649413592064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131544675225178 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129437501318144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2110712231936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110277368410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2107173907034 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131535570341888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7624143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17880619 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38464292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17293117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115420136233984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133646282573824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17293111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112469344608768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2950791625216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131535596733440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21171175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21149567 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115420052318208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112469316474368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133649413592064¥`w!—óO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17293117 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2110712231936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17293111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚ 950.615147268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10257037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133646282573824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131535596733440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133649413592064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131544675225178 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129437501318144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2110712231936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110277368410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2107173907034 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131535570341888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7624143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17880619 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38464292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17293117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115420136233984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133646282573824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17293111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112469344608768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2950791625216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13153559673344àÝöIÚ0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21171175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21149567 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115420052318208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112469316474368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133649413592064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21171175 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131547804114522 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚999 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21149567 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614MÙr0˜+ _wandbruntime‚951  _runtime‚951¦ãg[þ&™øM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115535496879616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2953685638656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131676181019494 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17898295 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7631005 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38481052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17310395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2953741406720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚ 951.51382998 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2112386161718 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133783020007424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129566923747328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38481052 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚78 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17310401 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55768064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131667090821632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112824545792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112824545792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17310401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112581727338496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129566898319360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21192291 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2115929185792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2109282700134 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112581755472896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38502692 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38502692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10267290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 % _timestamp‚1.7884289731581686e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131667090821632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17310395 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21170657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21170657 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2109282700134 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112581755472896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38502692 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38502692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10267290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 % _timestamp‚1.7884289731581686e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131667090821632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17310395 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21170657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21170657 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17898295 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7631005 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38481052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17310395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2953741406720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚ 951.51382998 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2112386161718 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133783020007424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129566923747328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38481052 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚78 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17310401 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55768064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131667090821632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112824545792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112824545792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17310401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112581727338496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129566898319360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21192291 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2115929185792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2215730614618 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8003424 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40364730 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138110301143040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138123107421530 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18157317 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884290197432623e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140326617090048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138110274751488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18157311 ZFDeviceStatsMonitor.on_train_£SÄ}Übatch_start/inactive_split_bytes.all.freed‚121187190450176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2219446965248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18772562 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40386344 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138119978532186 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22207419 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140329748108288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8002875 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2219446965248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 _runtime‚ 998.098583582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18157311 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140329748108288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22229027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10769690 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118089859680768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135907376806912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22207419 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140326617090048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10769687 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2212627153242 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121187274365952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135907351378944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18773114 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 trainer/global_step‚1049 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135907376806912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22207419 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140326617090048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10769687 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2212627153242 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121187274365952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135907351378944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18773114 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 trainer/global_step‚1049 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2215730614618 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8003424 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40364730 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138110301143040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55781376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138123107421530 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18157317 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884290197432623e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140326617090048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138110274751488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18157311 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121187190450176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2219446965248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18772562 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40386344 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138119978532186 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22207419 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140329748108288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8002875 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2219446965248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 _runtime‚ 998.098583582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18157311 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140329748108288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22229027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10769690 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118089859680768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984Qecòp0ª+ _wandbruntime‚998  _runtime‚998 Ȥ€ÿ&«ùM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136036773808128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40424744 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121302534164480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚82 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55768064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174595 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18790215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚1049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138241768839680 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚ 998.990442755 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2221559292416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18790743 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10779940 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40403104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218454652416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138241795231232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2221559292416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40424744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22228509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8010275 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174601 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138254613216054 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138241768839680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3100319387648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10779943 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8010800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118202270544896 $ _timestamp‚1.788429020634732e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121302618066944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138241795231232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40403104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140460223492096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22250143 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136036748380160aqzSجýM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55768064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174595 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18790215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚1049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138241768839680 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚ 998.990442755 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2221559292416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18790743 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10779940 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40403104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218454652416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138241795231232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2221559292416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40424744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22228509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8010275 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174601 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138254613216054 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138241768839680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83902464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3100319387648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10779943 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8010800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118202270544896 $ _timestamp‚1.788429020634732e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121302618066944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138241795231232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40403104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140460223492096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22250143 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136036748380160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136036773808128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40424744 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121302534164480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚82RUìµ0­+ _wandbruntime‚998  _runtime‚998Á2Õ¬Q®Ë 0loss/FBt1/assignment_loss‚0.4462372238905279 epoch‚0 trainer/global_step‚1049 /loss/FRt3/assignment_loss‚4.286866346995036 _runtime‚ 998.991759621 /loss/FRt1/assignment_loss‚3.827388252530779 /loss/FRt4/assignment_loss‚6.719296877200787 0loss/FBt3/assignment_loss‚0.6156234302432976 2loss/SRqqt1/assignment_loss‚1.9346568329767748 /loss/FRt2/detection_loss‚3.3558180584357333 0loss/SRqqt1/detection_loss‚1.784939288415692 1loss/SRqqt2/detection_loss‚4.1250717510347785 .loss/FBt3/detection_loss‚3.128499620101031 %loss/total_loss‚72.14494219973766 /loss/FRt1/detection_loss‚2.1945731842092107 .loss/FRt4/detection_loss‚3.261688702381574 0loss/SRqqt4/detection_loss‚2.896934086276639 2loss/SRqqt2/assignment_loss‚2.3954274447067925 1loss/FBt4/assignment_loss‚0.13394073483163793 /loss/FBt1/detection_loss‚13.057107529287626 -loss/FBt4/detection_loss‚2.27394958634457 _step‚83 1loss/SRqqt4/assignment_loss‚2.542246607042128 0loss/FBt2/assignment_loss‚0.7091209903464915 /loss/FRt3/detection_loss‚1.8466847244898479 1loss/SRqqt3/detection_loss‚1.6716842835671024 .loss/FBt2/detection_loss‚1.310737988559075 % _timestamp‚1.7884290206368048e+09 /loss/FRt2/assignment_loss‚5.587021295840923 1loss/SRqqt3/assignment_loss‚2.039427360029597Óï`U¯Ï 0loss/FBt1/assignment_loss‚0.4462372238905279 epoch‚0 trainer/global_step‚1049 /loss/FRt3/assignment_loss‚4.286866346995036 _runtime‚ 998.991759621 /loss/FRt1/assignment_loss‚3.827388252530779 /loss/FRt4/assignment_loss‚6.719296877200787 0loss/FBt3/assignment_loss‚0.6156234302432976 2loss/SRqqt1/assignment_loss‚1.9346568329767748 /loss/FRt2/detection_loss‚3.3558180584357333 0loss/SRqqt1/detection_loss‚1.784939288415692 1loss/SRqqt2/detection_loss‚4.1250717510347785 .loss/FBt3/detection_loss‚3.128499620101031 %loss/total_loss‚72.14494219973766 /loss/FRt1/detection_loss‚2.1945731842092107 .loss/FRt4/detection_loss‚3.261688702381574 0loss/SRqqt4/detection_loss‚2.896934086276639 2loss/SRqqt2/assignment_loss‚2.3954274447067925 1loss/FBt4/assignment_loss‚0.13394073483163793 /loss/FBt1/detection_loss‚13.057107529287626 -loss/FBt4/detection_loss‚2.27394958634457 _step‚83 1loss/SRqqt4/assignment_loss‚2.542246607042128 0loss/FBt2/assignment_loss‚0.7091209903464915 /loss/FRt3/detection_loss‚1.8466847244898479 1loss/SRqqt3/detection_loss‚1.6716842835671024 .loss/FBt2/detection_loss‚1.310737988559075 % _timestamp‚1.7884290206368048e+09 /loss/FRt2/assignment_loss‚5.587021295840923 1loss/SRqqt3/assignment_loss‚2.039427360029597S44ù‘°‚(:† â…åÔ¼ƒµµ gpu.0.gpu‚55.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚293.47' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Ä|QC±‚(:¸ â…åÔøøⶠdisk.rbd1.in‚0 disk.drbd1716.out‚0$ memory_percent‚92.80073642730713) proc.memory.availableMB‚ 1179.52734375$ disk.nvme0n1.out‚3153.9208984375 network.sent‚ 1.373554e+06* disk./.usagePercent‚17.838799039574425% disk./.usageGB‚265.81099700927734 disk.nvme0c0n1.in‚4.078125 disk.sda.in‚0 disk.drbd2265.out‚0 cpu‚25.073450792976793 disk.sda3.out‚36.71875 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.078125 disk.dm-0.in‚4.078125 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2953.22265625 proc.cpu.threads‚17! disk.dm-0.out‚3153.9208984375 disk.sda.out‚36.71875 disk.drbd2266.out‚0) proc.memory.percent‚18.02504062652588 disk.sda2.in‚0 network.recv‚419048 disk.rbd1.out‚ 0.61328125& disk.nvme0c0n1.out‚3153.9208984375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚36.71875rz“²‚(:ˆ ñ…åÔÌŸéµ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚128.362' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kÈõ…A³‚(:¶ ñ…åÔÏå϶ disk.sda.out‚37.34375$ memory_percent‚92.80226230621338# proc.memory.rssMB‚ 2953.22265625) proc.memory.percent‚18.02504062652588$ disk./.usageGB‚265.8385887145996 disk.rbd0.in‚0 disk.drbd2266.out‚0 network.recv‚425252& disk.nvme0c0n1.out‚3190.5205078125 disk.dm-0.in‚4.078125 disk.sda3.out‚37.34375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚37.34375$ disk.nvme0n1.out‚3190.5205078125 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 cpu‚25.07288275169332 network.sent‚ 1.423727e+06 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125) proc.memory.availableMB‚ 1179.27734375* disk./.usagePercent‚17.840650742069585! disk.dm-0.out‚3190.5205078125 disk.drbd2080.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0¥H@“´‚(:ˆ €†åÔ«á…µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^yíÎEµ‚(:º €†åÔ²¥È¶ disk.sda.in‚0 disk.sda3.out‚ 37.703125& disk.nvme0c0n1.out‚3224.1708984375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0! disk.dm-0.out‚3224.1708984375 cpu‚25.072561739859758 disk.sda.out‚ 37.703125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0$ memory_percent‚92.80076026916504 proc.cpu.threads‚17 disk.nvme0n1.in‚4.078125( proc.memory.availableMB‚ 1179.5234375% disk./.usageGB‚265.84478759765625$ disk.nvme0n1.out‚3224.1708984375 disk.sda2.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2953.22265625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125 network.sent‚ 1.429052e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 37.703125 network.recv‚428846* disk./.usagePercent‚17.841066754312724 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125) proc.memory.percent‚18.02504062652588ˆØ®`2¶- _wandbruntime‚1045  _runtime‚1045¬íÌþÈ·Â _step‚84 ' lr-AdamW/pg1‚0.00018595773270176547 ' lr-AdamW/pg2‚0.00018595773270176547 trainer/global_step‚1099 % _timestamp‚1.7884290671350784e+09 _runtime‚1045.490118437»c¯XÌ¸Æ ' lr-AdamW/pg2‚0.00018595773270176547 trainer/global_step‚1099 % _timestamp‚1.7884290671350784e+09 _runtime‚1045.490118437 _step‚84 ' lr-AdamW/pg1‚0.00018595773270176547Tmt#Û2¹- _wandbruntime‚1045  _runtime‚1045yòŒaö'ºðO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11282337 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19021517 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144695281839194 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚85 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8381543 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23286879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42308396 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2318080399450 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144684979161088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2325077071872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19663340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11282340 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126954486019584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147006951606272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144698410728538 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 trainer/global_step‚1099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19021517 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147006951606272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2325077071872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2321183860826 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23265271 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2321972445184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3244001313792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144685005552640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128889344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144685005552640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8381003 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123710402887168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _runtime‚1045.491117102 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19663883 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23286879 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19663883 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81818624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23286879 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8381543 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23286879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42308396 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2318080399450 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144684979161088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2325077071872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19663340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11282340 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126954486019584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147006951606272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144698410728538 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 trainer/global_step‚1099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19021517 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147006951606272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚53684224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2325077071872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876685548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2321183860826 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23265271 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2321972445184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.sma˜û¨oÙll_pool.freed‚ 3244001313792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144685005552640Uñ(22¼- _wandbruntime‚1046  _runtime‚10464/'½úM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144816473249280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 _runtime‚ 1046.41606728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2323292654134 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2320189192550 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23307995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19038801 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144816499640832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2324084759040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19038795 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19038795 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127069836571136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42325156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147143689039872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144829916523062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144816473249280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42325156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127069754765824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123822813751296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144826787633510 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19681233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3246941014528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚1099 % _timestamp‚1.7884290680605328e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2324084759040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23286361 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8388643 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2327189399040 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142506623868928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚86 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42346796 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144816499640832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42346796 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552ä ‹'¾þM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144829916523062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131031552 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144816473249280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42325156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127069754765824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123822813751296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144826787633510 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19681233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3246941014528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚1099 % _timestamp‚1.7884290680605328e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2324084759040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23286361 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8388643 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2327189399040 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142506623868928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚86 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3104640000 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42346796 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144816499640832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42346796 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚53670912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144816473249280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 _runtime‚ 1046.41606728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2323292654134 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2320189192550 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23307995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19038801 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144816499640832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2324084759040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19038795 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19038795 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81805312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3877865472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127069836571136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42325156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147143689039872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968Võ†±x2¿- _wandbruntime‚1046  _runtime‚1046 ÊSÀÍ .loss/FBt4/detection_loss‚1.425839411664047 .loss/FRt1/detection_loss‚2.071638641376344 .loss/FRt2/detection_loss‚2.805542852729559 /loss/FRt1/assignment_loss‚4.221348412453183 1loss/SRqqt2/assignment_loss‚2.860368695014562 1loss/FBt1/assignment_loss‚0.40470685369014064 /loss/FRt3/assignment_loss‚3.976004739196933 2loss/SRqqt1/assignment_loss‚2.3576867402489508 .loss/FBt1/detection_loss‚4.311902791075409 _step‚87 /loss/FRt2/assignment_loss‚5.544632516801357 1loss/SRqqt3/detection_loss‚1.6261528620594425 .loss/FRt4/detection_loss‚4.770789077648749 %loss/total_loss‚66.92101949224215 % _timestamp‚1.7884290680625188e+09 _runtime‚1046.417495175 1loss/SRqqt1/detection_loss‚1.5893816358562727 trainer/global_step‚1099 2loss/SRqqt3/assignment_loss‚1.8620262369513512 1loss/SRqqt4/assignment_loss‚2.503178368354666 1loss/FBt2/assignment_loss‚0.26440958731691355 1loss/FBt4/assignment_loss‚0.41247982303299174 /loss/FRt3/detection_loss‚2.4334468336737887 0loss/FBt3/assignment_loss‚0.1988538709384448 /loss/FRt4/assignment_loss‚7.349284318777231 .loss/FBt2/detection_loss‚4.400666895508766 epoch‚0 0loss/SRqqt4/detection_loss‚2.903190177062462 1loss/SRqqt2/detection_loss‚2.6850307132953253 .loss/FBt3/detection_loss‚3.942457437515259)xæ´WÁÑ 1loss/SRqqt2/detection_loss‚2.6850307132953253 .loss/FBt3/detection_loss‚3.942457437515259 .loss/FBt4/detection_loss‚1.425839411664047 .loss/FRt1/detection_loss‚2.071638641376344 .loss/FRt2/detection_loss‚2.805542852729559 /loss/FRt1/assignment_loss‚4.221348412453183 1loss/SRqqt2/assignment_loss‚2.860368695014562 1loss/FBt1/assignment_loss‚0.40470685369014064 /loss/FRt3/assignment_loss‚3.976004739196933 2loss/SRqqt1/assignment_loss‚2.3576867402489508 .loss/FBt1/detection_loss‚4.311902791075409 _step‚87 /loss/FRt2/assignment_loss‚5.544632516801357 1loss/SRqqt3/detection_loss‚1.6261528620594425 .loss/FRt4/detection_loss‚4.770789077648749 %loss/total_loss‚66.92101949224215 % _timestamp‚1.7884290680625188e+09 _runtime‚1046.417495175 1loss/SRqqt1/detection_loss‚1.5893816358562727 trainer/global_step‚1099 2loss/SRqqt3/assignment_loss‚1.8620262369513512 1loss/SRqqt4/assignment_loss‚2.503178368354666 1loss/FBt2/assignment_loss‚0.26440958731691355 1loss/FBt4/assignment_loss‚0.41247982303299174 /loss/FRt3/detection_loss‚2.4334468336737887 0loss/FBt3/assignment_loss‚0.1988538709384448 /loss/FRt4/assignment_loss‚7.349284318777231 .loss/FBt2/detection_loss‚4.400666895508766 epoch‚0 0loss/SRqqt4/detection_loss‚2.903190177062462Wv¢]ד‚(:ˆ †åÔ§š¡µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚265.462' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì5ŽqHÂ(:½ †åÔ‰œÞ· disk.rbd0.out‚0) proc.memory.percent‚18.02504062652588 disk.dm-1.out‚ 38.64453125 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚3257.2841796875 disk.dm-0.in‚4.078125 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 network.sent‚ 1.435063e+06% disk./.usageGB‚265.88659286499023 disk.sda2.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0# memory_percent‚92.7968978881836& proc.memory.availableMB‚ 1180.15625 disk.nvme0c0n1.in‚4.078125 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 38.64453125 disk.drbd2265.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚3257.2841796875# proc.memory.rssMB‚ 2953.22265625 cpu‚25.072190786309797 proc.cpu.threads‚17 network.recv‚433201 disk.drbd2266.out‚0* disk./.usagePercent‚17.843872340880463 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.078125$ disk.nvme0n1.out‚3257.2841796875 disk.sda3.out‚ 38.64453125™ôh“Ä‚(:ˆ ž†åÔÙÌê´ gpu.0.gpu‚87.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚282.912' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢øýHÅ‚(:½ ž†åԯ⶷ disk.dm-0.in‚4.078125 disk.sda3.out‚ 39.2265625 disk.drbd1716.in‚0 disk.drbd2080.out‚0% disk.nvme0n1.out‚3308.50439453125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0' disk.nvme0c0n1.out‚3308.50439453125" disk.dm-0.out‚3308.50439453125 disk.drbd2080.in‚0 disk.sda.out‚ 39.2265625 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0 disk.rbd1.in‚0 cpu‚25.0716707584391 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 39.2265625 disk.rbd1.out‚ 0.61328125 network.sent‚ 1.484816e+06 network.recv‚438409* disk./.usagePercent‚17.844169565627716 disk.sda.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0$ memory_percent‚92.79694557189941( proc.memory.availableMB‚ 1180.1484375 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2953.22265625) proc.memory.percent‚18.02504062652588$ disk./.usageGB‚265.8910217285156 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚4.078125_+³3“Æ‚(:ˆ ­†åÔ„ö³µ gpu.0.gpu‚43.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚287.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÀbMÇ‚(: ­†åÔØç¶ disk.drbd2080.out‚0' disk.nvme0c0n1.out‚3344.33251953125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0) proc.memory.percent‚18.02504062652588 network.recv‚443330 disk.rbd1.out‚ 0.61328125 disk.dm-0.in‚4.078125" disk.dm-0.out‚3344.33251953125 disk.sda.out‚ 39.74609375 disk.rbd0.out‚0 disk.sda2.in‚0 network.sent‚ 1.491066e+06% disk./.usageGB‚265.88664627075195 disk.rbd1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1179.89453125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 39.74609375 disk.drbd2266.in‚0 disk.sda.in‚0 disk.rbd0.in‚0$ memory_percent‚92.79849529266357# proc.memory.rssMB‚ 2953.22265625) disk./.usagePercent‚17.84387592498594% disk.nvme0n1.out‚3344.33251953125 cpu‚25.07140583337746 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0 disk.sda3.out‚ 39.74609375 disk.drbd2080.in‚0 disk.drbd2265.out‚0¡ùK2È- _wandbruntime‚1092  _runtime‚1092¦n,¸ÈÉ trainer/global_step‚1149 % _timestamp‚1.7884291145722094e+09 _runtime‚1092.927248346 _step‚88 ' lr-AdamW/pg1‚0.00019441804811130893 ' lr-AdamW/pg2‚0.00019441804811130893|…aaÌÊÆ _step‚88 ' lr-AdamW/pg1‚0.00019441804811130893 ' lr-AdamW/pg2‚0.00019441804811130893 trainer/global_step‚1149 % _timestamp‚1.7884291145722094e+09 _runtime‚1092.927248346XÕö`2Ë- _wandbruntime‚1092  _runtime‚1092âR/Ù ÌñO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44230448 TGDeviceStatsMonitor.on_train_batch_start/allocation.lÞn Øîarge_pool.allocated‚19885717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153687286122496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44208834 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19885711 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2427602551808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151273714035546 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151270585146202 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11794987 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44230448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19885717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153687286122496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131018240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83915776 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44208834 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19885711 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26466 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2427602551808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151273714035546 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151270585146202 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11794987 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151391177658880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153824023556096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24344213 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚90 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44247208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11805243 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151402090940518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129443356957696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44247208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3393560253952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44268848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148976473929728 trainer/global_step‚1149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132836859346432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8767713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129443385092096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151391177658880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8767177 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2429714865664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2429714865664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153820892524544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1093.825057926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19902995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24365847 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132836945346048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20572417 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57865216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44268848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2432819505664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129443356957696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44247208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3393560253952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44268848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148976473929728 trainer/global_step‚1149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132836859346432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8767713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129443385092096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151391177658880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21634 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8767177 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2429714865664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2429714865664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153820892524544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1093.825057926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19902995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24365847 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132836945346048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20572417 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57865216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44268848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2432819505664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151391177658880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153824023556096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24344213 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚90 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44247208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11805243 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151402090940518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Z™ï2Ñ- _wandbruntime‚1093  _runtime‚1093w8‘?XÒÒ 2loss/SRqqt3/assignment_loss‚1.9672148531346128 .loss/FBt2/detection_loss‚6.164944100379944 1loss/SRqqt2/detection_loss‚2.9513775888950593 % _timestamp‚1.7884291154712791e+09 0loss/FBt4/assignment_loss‚0.4127464246076194 /loss/FRt2/detection_loss‚2.6011788094485246 /loss/FRt4/assignment_loss‚6.968697800355799 2loss/SRqqt2/assignment_loss‚2.6688546480671054 2loss/SRqqt4/assignment_loss‚2.8634189234839544 /loss/FRt4/detection_loss‚3.7802709691664753 1loss/SRqqt1/detection_loss‚1.5824611292531092 .loss/FBt3/detection_loss‚5.753300453935351 1loss/FBt1/assignment_loss‚0.42745572608406895 -loss/FRt3/detection_loss‚2.27282267865145 %loss/total_loss‚67.47351683477152 epoch‚0 /loss/FRt1/assignment_loss‚3.772236845949117 /loss/FRt1/detection_loss‚1.8972862108665354 1loss/SRqqt4/detection_loss‚2.9844658551392733 _runtime‚1093.826302987 1loss/FBt3/assignment_loss‚0.29485408389674767 /loss/FBt1/detection_loss‚2.6256795028845468 .loss/FBt4/detection_loss‚2.205870284273541 0loss/FRt2/assignment_loss‚5.6358457847877785 /loss/FRt3/assignment_loss‚4.023169893138814 1loss/FBt2/assignment_loss‚0.46516679654450854 _step‚91 2loss/SRqqt1/assignment_loss‚1.7870264508657985 trainer/global_step‚1149 0loss/SRqqt3/detection_loss‚1.367171020961778‘DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12307640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20749917 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25402583 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21446427 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134951489299968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 % _timestamp‚1.7884291618605204e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25380975 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20749917 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160370751656960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155316901561344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134951517434368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3536934171648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3536874195968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20749911 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9138213 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155316926989312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157834387980288 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚88110080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138488363495936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157845888453210 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46152500 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9138787 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160367620638720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12307637 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157834387980288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160370751656960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25402583 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _step‚93 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157834414371840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2536337285120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138488451606016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚574 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59975680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160367620638720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25380975 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2528986891866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160370751656960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3104626688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155316901561344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134951517434368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3536934171648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3536874195968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20749911 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9138213 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155316926989312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645471701 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157834387980288 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚88110080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138488363495936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157845888453210 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3104626688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46152500 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9138787 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160367620638720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12307637 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157834387980288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160370751656960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25402583 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _step‚93 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157834414371840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2536337285120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138488451606016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚574 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59975680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160367620638720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25380975 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2528986891866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26466 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3877865472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12307640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20749917 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25402583 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131018240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21446427 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134951489299968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103461376 % _timestamp‚1.7884291618605204e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25380975 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20749917 XIDeviceStatsMonitor.on_train_batchÙŽéw_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3877865472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0]úèäø2â- _wandbruntime‚1141  _runtime‚1141\Ž*á'ãúM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1199 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2538449612288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2534199146550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135063900164096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21463773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20767195 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157965882068480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155446298562560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25423699 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2535344972288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46169260 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46190900 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157965908460032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157965882068480 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2538449612288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138603783876096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20767201 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚88096768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46190900 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3539795615232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157977394247526 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25423699 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46169260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9146428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138603695779328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20767201 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12317893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12317890 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2531095684966 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160504358072320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878K+ët'äþM PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128889552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2535344972288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46169260 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46190900 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157965908460032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157965882068480 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2538449612288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138603783876096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21634 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20767201 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚88096768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645471701 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46190900 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3539795615232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157977394247526 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25423699 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46169260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3104640000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9146428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138603695779328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20767201 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103461584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12317893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12317890 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2531095684966 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26466 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160504358072320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3877865472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1199 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2538449612288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2534199146550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876685548 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135063900164096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21463773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20767195 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157965882068480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155446298562560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25423699 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131031552^)Žâ2å- _wandbruntime‚1141  _runtime‚1141Q´@…VæÐ trainer/global_step‚1199 2loss/SRqqt4/assignment_loss‚2.2594282987217107 0loss/SRqqt4/detection_loss‚3.220823826889197 /loss/FBt1/detection_loss‚1.8404577647202782 /loss/FRt4/assignment_loss‚6.167902329388787 .loss/FBt2/detection_loss‚4.740184418857098 .loss/FBt3/detection_loss‚3.271304849535227 .loss/FBt4/detection_loss‚4.497119337698338 %loss/total_loss‚62.61054312275557 .loss/FRt4/detection_loss‚3.850613650153665 1loss/SRqqt2/detection_loss‚2.6693690579671125 % _timestamp‚1.7884291627652247e+09 0loss/FRt3/assignment_loss‚3.7942890058864247 1loss/SRqqt1/assignment_loss‚2.088627536507214 1loss/FBt1/assignment_loss‚0.32885904028144836 0loss/FBt4/assignment_loss‚0.3932012663497515 _runtime‚1141.120185469 2loss/SRqqt2/assignment_loss‚2.3471316511814413 0loss/FBt2/assignment_loss‚0.6038229718493918 1loss/FBt3/assignment_loss‚0.13952571019586024 /loss/FRt1/detection_loss‚2.3419940492101743 /loss/FRt2/detection_loss‚2.6505255968339982 /loss/FRt3/detection_loss‚2.1471865984526546 1loss/SRqqt1/detection_loss‚1.3957161022459759 1loss/SRqqt3/detection_loss‚1.3743560950943203 /loss/FRt2/assignment_loss‚4.732028599708311 1loss/SRqqt3/assignment_loss‚1.868661493972637 _step‚95 /loss/FRt1/assignment_loss‚3.887413871054556 epoch‚0Ü ¥ZçÔ /loss/FRt4/assignment_loss‚6.167902329388787 .loss/FBt2/detection_loss‚4.740184418857098 .loss/FBt3/detection_loss‚3.271304849535227 .loss/FBt4/detection_loss‚4.497119337698338 %loss/total_loss‚62.61054312275557 .loss/FRt4/detection_loss‚3.850613650153665 1loss/SRqqt2/detection_loss‚2.6693690579671125 % _timestamp‚1.7884291627652247e+09 0loss/FRt3/assignment_loss‚3.7942890058864247 1loss/SRqqt1/assignment_loss‚2.088627536507214 1loss/FBt1/assignment_loss‚0.32885904028144836 0loss/FBt4/assignment_loss‚0.3932012663497515 _runtime‚1141.120185469 2loss/SRqqt2/assignment_loss‚2.3471316511814413 0loss/FBt2/assignment_loss‚0.6038229718493918 1loss/FBt3/assignment_loss‚0.13952571019586024 /loss/FRt1/detection_loss‚2.3419940492101743 /loss/FRt2/detection_loss‚2.6505255968339982 /loss/FRt3/detection_loss‚2.1471865984526546 1loss/SRqqt1/detection_loss‚1.3957161022459759 1loss/SRqqt3/detection_loss‚1.3743560950943203 /loss/FRt2/assignment_loss‚4.732028599708311 1loss/SRqqt3/assignment_loss‚1.868661493972637 _step‚95 /loss/FRt1/assignment_loss‚3.887413871054556 epoch‚0 trainer/global_step‚1199 2loss/SRqqt4/assignment_loss‚2.2594282987217107 0loss/SRqqt4/detection_loss‚3.220823826889197 /loss/FBt1/detection_loss‚1.8404577647202782_(3hƒ’è‚(:‡ ø†åÔ·¾‘µ gpu.0.gpu‚31.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÒ4ïGé‚(:¼ ø†åÔ´¡é· disk.drbd2266.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 42.1015625 disk.sda2.in‚0 disk.drbd1716.in‚0( disk./.usagePercent‚17.8486809303961$ disk.nvme0n1.out‚3584.3291015625 disk.sda3.in‚0 disk.sda3.out‚ 42.1015625 disk.drbd2080.out‚0" proc.memory.rssMB‚ 3025.5859375 network.recv‚464714 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 42.1015625 cpu‚25.089057504636216 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125$ memory_percent‚93.23737621307373% disk./.usageGB‚265.95824432373047 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0& disk.nvme0c0n1.out‚3584.3291015625! disk.dm-0.out‚3584.3291015625 disk.drbd2080.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1107.98828125 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.466711044311523 network.sent‚ 1.563887e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0Äï"Uœê‚(:‘ ‡‡åÔ˸µ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.004200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Š²Lë‚(:Á ‡‡åÔ”ƒ· disk.dm-1.in‚0 network.sent‚ 1.613938e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 disk.drbd2266.in‚0# proc.memory.rssMB‚ 3026.07421875 proc.cpu.threads‚17% disk./.usageGB‚265.95565032958984& disk.nvme0c0n1.out‚3620.7236328125 disk.dm-0.in‚4.078125! disk.dm-0.out‚3620.7236328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚3620.7236328125 disk.sda3.out‚ 42.76953125 disk.sda.in‚0* proc.memory.percent‚18.469691276550293 disk.dm-1.out‚ 42.76953125 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚93.24066638946533) proc.memory.availableMB‚ 1107.44921875* disk./.usagePercent‚17.848506845272816 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 42.76953125 disk.rbd1.in‚0 disk.sda3.in‚0 cpu‚25.13814834418367 disk.sda2.out‚0 network.recv‚470348 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125 disk.nvme0c0n1.in‚4.078125Géå­“ì‚(:ˆ –‡åÔ’¬­µ gpu.0.gpu‚98.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ GOí‚(:Ä –‡åԙ̌· disk.nvme0c0n1.in‚4.078125) proc.memory.availableMB‚ 1106.91015625 proc.cpu.threads‚17 network.recv‚474309 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.61328125 disk.dm-0.in‚4.078125 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚3663.02197265625' disk.nvme0c0n1.out‚3663.02197265625 disk.sda3.out‚ 43.31640625 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚3663.02197265625* proc.memory.percent‚18.471097946166992 cpu‚25.193221352783887 disk.drbd2080.out‚0 disk.dm-1.out‚ 43.31640625$ memory_percent‚93.24395656585693" proc.memory.rssMB‚ 3026.3046875 network.sent‚ 1.619822e+06* disk./.usagePercent‚17.850270225168664 disk.drbd2266.out‚0 disk.rbd1.in‚0% disk./.usageGB‚265.98192596435547 disk.drbd2080.in‚0 disk.sda.out‚ 43.31640625 disk.nvme0n1.in‚4.078125ÉÙ3~’î‚(:‡ ¥‡åÔ²ºµ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚122.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ïsçEï‚(:º ¥‡åԼɥ· disk.rbd0.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.471169471740723 disk.dm-1.in‚0 disk.sda3.out‚ 43.859375 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.078125 disk.sda2.in‚0 disk.dm-0.in‚4.078125 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.078125' disk.nvme0c0n1.out‚3703.68896484375 disk.drbd2265.out‚0 network.sent‚ 1.626131e+06% disk./.usageGB‚265.99232482910156 disk.drbd1716.out‚0$ memory_percent‚93.24464797973633# proc.memory.rssMB‚ 3026.31640625" disk.dm-0.out‚3703.68896484375 disk.drbd2080.out‚0 cpu‚25.243469643828547 proc.cpu.threads‚17 network.recv‚478984' disk./.usagePercent‚17.850968101707 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 43.859375 disk.rbd0.out‚0% disk.nvme0n1.out‚3703.68896484375' proc.memory.availableMB‚ 1106.796875 disk.dm-1.out‚ 43.859375 disk.sda3.in‚0 disk.drbd1716.in‚08™Rkvðbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPgƒ%hñbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JPI †\fòba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPÒ·}Ïtóbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP¸µôtôbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP”ª wpõbk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÑsélöbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP¥ÀRf÷ba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP*rÂmøbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JP\Þmùbh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JPäkŠâvúbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP¼4Úe]ûbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JPûÀk¬rübm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPû§ˆøuýbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP.)ùyþbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPb>;oÿbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JP êt€bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP Ÿ obj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPþZN§w‚br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP<ªgørƒbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP ÓHÖr„bm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JPôEYNr…bm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPã:Kqp†bk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JPÌŒN…o‡bj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JP|ãwdˆb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329204224 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚21569454 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353498624 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚166121271376384 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚163539368839168 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚166121271376384 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329204224 _runtime‚1204.765670988 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚163539368839168 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚163350947843646 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351018380 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329204224 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚21569454 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353498624 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚166121271376384 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚163539368839168 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚166121271376384 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329204224 _runtime‚1204.765670988 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚163539368839168 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚163350947843646 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351018380 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPÀçkïjöbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPÞ út÷bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP‹–RUføba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP´ü0vùbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP7ÞIÁzúbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPhí}µjûbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP§wåõvübq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPä>Òucýb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JPN Fkþbf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPYû´Wgÿbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPGÄ/‚m€bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JP2­0,pbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JP_,¡,x‚bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPMK6Ìjƒbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPð_Þu„bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP³±Gn…bi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPË™œ¶p†bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPâ^ŒÉ|‡bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP‰q™ezˆbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPµJØh‰bc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JP¾ëgŠbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP_ƒ»wp‹bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÀAD®pŒbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPt±¤ˆubp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPf‘ äuŽbp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP|‡X­mbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JP%} vbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPã$¦i‘bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPð¢“òr’bm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP ¼ÝÜn“bi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPF*d¶c”b^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP¾×ô5d•b_ iÀbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JP­"“wÁbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPèÒÖqÂbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP@’+3mÃbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JP“1¦ãtÄbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP ií)jÅbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP£ Ç­oÆbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JPÅ7ÜlÇbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP9kˆrÈbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP ÇAëuÉbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPg!:¹kÊbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JP£EËrËbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPè€jÔwÌbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP±®sÍbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP¥Ñ¿qÎbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP±Á<@mÏbh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JPH ŽqÐbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP$s[[ÑbV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP_¶8%rÒbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPB*~gÓbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPÄe,lÔbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPõ­®©rÕbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP2>pÖbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP"1Uûs×bn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP­I.lØbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JP¨RüDkÙbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPu¹8lÚbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JP´á}rÛbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPµÖÞInÜbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPK¹XOpÝbk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP§³ R|Þbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPÉ0lßbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPU"¶,jàbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP>Œ„%tábo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPgp­mâbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPuÆØvãbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP—†¦aeäb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP:ÌKÒnåbi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JP¦5˜kæbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP.O kçbf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JPñŠþrèbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPÎÖÒl2é- _wandbruntime‚1205  _runtime‚1205™}>i)êãR VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚9450383 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚21582657 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚21582652 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚166211362003456 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚163628961448960 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚25998162 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚25980670 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚163628985743360 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚26719 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚9450498 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚47580819 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚47563322 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚17492 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3591719085056 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚25998162 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2580047055872 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚139839462338048 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚97200640 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚47580819 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚163628961448960 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚166211362003456 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329204224 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚143431278623744 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚21582657 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚25980670 _runtime‚1205.819170802 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚47563322 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚25980670 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚163628985743360 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚26719 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚9450498 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚47580819 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚47563322 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚17492 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3591719085056 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚25998162 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2580047055872 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚139839462338048 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚97200640 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚47580819 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚163628961448960 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚166211362003456 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329204224 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚143431278623744 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚21582657aãH ì‚(:’ ´‡åÔê–þµ gpu.0.gpu‚19.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.202' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.440400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06„XOí‚(:Ä ´‡åÔúÞ· disk.drbd1716.out‚0% disk.nvme0n1.out‚3750.74169921875 disk.sda2.in‚0 disk.sda.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.078125 network.recv‚482408 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125" disk.dm-0.out‚3750.74169921875* disk./.usagePercent‚17.852584021263127 disk.rbd1.in‚0 cpu‚25.291712399152093 disk.sda3.out‚ 44.22265625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 44.22265625' disk.nvme0c0n1.out‚3750.74169921875* proc.memory.percent‚18.471455574035645 disk.rbd1.out‚ 0.61328125 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3026.36328125 proc.cpu.threads‚17$ disk./.usageGB‚266.0164031982422 disk.sda.out‚ 44.22265625 disk.drbd2266.in‚0$ memory_percent‚93.24438571929932 network.sent‚ 1.631368e+06 disk.sda1.in‚0 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1106.83984375 disk.drbd1716.in‚0éTšé@îb; particle/accuracy_4_of_5"trainer/global_step2:JP#îs{@ïb; particle/accuracy_1_of_6"trainer/global_step2:JPpæ6à;ðb6 jet/accuracy_1_of_7"trainer/global_step2:JPþÓ–JñbE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP¾H'NòbI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¨‡”HóbC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPºy‡˜JôbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP͉;õb6 jet/accuracy_2_of_7"trainer/global_step2:JP¹ËHöbC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPš’Ž_H÷bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÅ]¦ÖJøbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPñÅåNùbI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP¤oìHúbC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPܶ‚åJûbE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP ¼ùŠJübE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÙᙈJýbE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPNQZåNþbI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP;±SJÿbE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¥3G>N€bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPYd@b; particle/accuracy_3_of_5"trainer/global_step2:JPQo><‚b7 particle/sensitivity"trainer/global_step2:JP{áNƒbI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPrkÜÃH„bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPyçBH…bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPên²SN†bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÀ°Ð‚H‡bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP«ûÐHˆbC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP<9N‰bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPu†˜§JŠbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPƒ|}H‹bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP’Ø(@Œb; particle/accuracy_3_of_6"trainer/global_step2:JP L4JbE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP5£"gJŽbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPû!NbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP<¸W}JbE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÛýDXJ‘bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP ›Ü*J’bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP­ŽözH“bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPqf¿kJ”bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¼:qlN•bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPAþT¼N–bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP5A`÷N—bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP(ý¦’J˜bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP„å3uN™bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÏéNšbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP Ó»N›bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP^ *GJœbE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPbVÂJbE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP¨|a‹JžbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPƒ¢»ÂHŸbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPD’;ƒN bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP x‡yN¡bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPâ§B§J¢bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŸéApN£bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP@1]H¤bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP„{ÁJ¥bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP׎*N¦bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPiºJ§bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPŽ«_N¨bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPIÓ kJ©bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPMhðÉHªbC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPd+(2N«bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPd;•¹J¬bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP‘ÖrJ­bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP^´taN®bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPw£íN¯bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP1„òÞJ°bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPê>çH±bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPYWh#J²bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÚ¤ÒN³bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¾rH´bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPµmI€HµbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP%{mJÒbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP+§çýHÓbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÍ-JÔbE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP¡0 HÕbC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP€”º^HÖbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPŠß‰êJ×bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPæ¡èJØbE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPV^kGJÙbE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP×NöNÚbI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP jíNÛbI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPhp·HÜbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÉ|EÑNÝbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPô0JÞbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP‚/DÆHßbC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP~ UJàbE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP2hâJábE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP|\`GHâbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¸<¶NãbI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP{kòüJäbE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPÊùŸHåbC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPÜ«JJæbE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÙ%z•JçbE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP¬¨¸JèbE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPdäòÓHébC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPgiLJêbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP6ZDHëbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPƒÆ ´HìbC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPhHíbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP—ee4JîbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPýê¼NïbI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP…4ØXNðbI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPé•©æ;ñb6 jet/accuracy_4_of_4"trainer/global_step2:JP£ÓµHòbC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPnõÑ.JóbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP!Í4JôbE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPý¸pHõbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPU3:ôHöbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP) gJ÷bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP.Ô•›JøbE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPèh}£HùbC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPFNÿðHúbC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP#€­«JûbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP:Ø¢@NübI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPyiÝHýbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP~¹mçJþbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP‚-…NÿbI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP»þlvJ€ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP´´Œê; b6 jet/accuracy_1_of_5"trainer/global_step2:JP@_…¾@‚ b; particle/accuracy_1_of_5"trainer/global_step2:JPl× Hƒ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPƒç¥N„ bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÀêÔÑN… bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP Ú±#H† bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPí¥ô;‡ b6 jet/accuracy_6_of_7"trainer/global_step2:JPöË Hˆ bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPx'œ™J‰ bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPéþNŠ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP /ryJ‹ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPëñ©±HŒ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒo§J bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPKWÔâNŽ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÐœõN bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPHE8 b3 particle/f_score"trainer/global_step2:JP“|T—;‘ b6 loss/val_total_loss"trainer/global_step2:JPàGbYJ’ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¤¯^J“ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP[õõÒH” bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP;³Ï¹N• bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPh@AíN– bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPƒ Ò;J— bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPô}q‚H˜ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPDbŦH™ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPXØ5¢;š b6 jet/accuracy_1_of_3"trainer/global_step2:JP4Us™J› bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÉ*ÂúJœ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPV8¯QJ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPÌKÔJž bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÝ*·NŸ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP¤Äs÷J  bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÃCagN¡ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPˆPÙH¢ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP3¦èF@£ b; particle/accuracy_2_of_6"trainer/global_step2:JPñŠ$hJ¤ bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP½BdØH¥ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPm®w™N¦ bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP½Ç{™N§ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPꥥJ¨ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP|ký5N© bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPd¨ÚNª bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP8ÊUN« bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPK±N¬ bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÕ,Ï‘J­ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPˆq4/H® bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPçºÖ;¯ b6 jet/accuracy_5_of_6"trainer/global_step2:JP“Îw•J° bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPV^L0N± bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÛà¾3H² bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPáù#£J³ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPkA8ìN´ bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPãQ’¶Jµ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPö(‹º;¶ b6 jet/accuracy_4_of_7"trainer/global_step2:JPúr»ŠN· bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPƒjïµN¸ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPµ!û"J¹ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP¸LSoHº bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP ÈJè bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP]e&ØNé bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPI¿%4Hê bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPfC¨Á;ë b6 jet/accuracy_1_of_6"trainer/global_step2:JPsÖÈHì bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP™Y÷Jí bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPrÎJî bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP…G,ÏJï bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¢XP~Jð bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP fØNñ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP‹Nò bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPAvÚJó bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPEôTHô bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP|å-ƒJõ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP;ïÖ!Hö bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP©í°H‘ bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPIB9 N’ bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÄ¡‘šN“ bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÛÇÉJ” bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPH½X7J• bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¡LH– bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP;¨t N— bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP»,,N˜ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP0nÞJ™ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP.`¸Jš bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPù¤×xH› bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPAëHHœ bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPþûØH bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP{‰–ûJž bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP*Êtî@Ÿ b; particle/accuracy_6_of_7"trainer/global_step2:JPÀnÈMJ  bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPõò GJ¡ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPy`eH¢ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP(ýÔ¡N£ bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPðÆÜJ¤ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPBœáÆJ¥ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP+ÃñÛN¦ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÃ8Ñ-N§ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÕ°yÔN¨ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPB7uUJ© bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPSr˜öHª bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPuŸhPN« bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP;·ËH¬ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPfœóTN­ bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPrmØ;® b6 jet/accuracy_5_of_5"trainer/global_step2:JP@Ö×P9¯ b4 particle/accuracy"trainer/global_step2:JPWÙñH° bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÔ'Ò˜;± b6 jet/accuracy_4_of_6"trainer/global_step2:JP8ÆH² bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP ¨ýOH³ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP„V7N´ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP ”äÐJµ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP ‡ uH¶ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÈd1J· bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPJi7ÆH¸ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPK©ëN¹ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP¿ççïHº bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPTè-J» bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPjcœSN¼ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPóßJ½ bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPñg‚N¾ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP½>¶@¿ b; particle/accuracy_7_of_7"trainer/global_step2:JPbÓÉNÀ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPµ2ò~NÁ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÿË ýJ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÏß¡Hà bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPwl&HÄ bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPA-Â;Å b6 jet/accuracy_2_of_4"trainer/global_step2:JPqaûü@Æ b; particle/accuracy_2_of_4"trainer/global_step2:JP-•ØEJÇ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÄ?=NÈ bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP€õ§ÒHÉ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPôĉJÊ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPéÊLÜNË bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP^G(3JÌ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPí›iãÍ bC Purity/1FRt3SRqäß÷ 2qt2FBt/FRt_purity"trainer/global_step2:JPÀ.Þ"JÎ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP»àAJÏ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP:},¾HÐ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP½„\JÑ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP[TŸˆJÒ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP ·NÓ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP‰¸®HÔ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÒº˜ÑJÕ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP›HÖ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP Äl×J× bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPZJØ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPV±eìNÙ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÃo­NÚ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP-ÞV&JÛ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPd¥JÜ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPNeõjNÝ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP7ÙcƒJÞ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP:P»sHß bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP÷ ¨Jà bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÃê´ãNá bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP‹*ÑFJâ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP¦Íj!Jã bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPä«fdHä bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP•„á–Jå bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP„µÑRNæ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP2SÀ7Hç bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPŠÉNè bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPö Jé bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP «–Hê bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP="ÌuHë bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPy{GrJì bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP}B‚ÞHí bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP熻0Hî bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPŠ†R§Hï bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆFuHð bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPp_ QJñ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP´\AHò bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPDhÙ‹Hó bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP¦ð›Jô bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPsGà³Nõ bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP‹ç{Nö bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPvJ÷ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPmÓƒ¦Nø bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPA®i@ù b; particle/accuracy_1_of_2"trainer/global_step2:JP½ÁÔH;ú b6 jet/accuracy_3_of_6"trainer/global_step2:JPú˜”Jû bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPu Nü bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP•Ï TNý bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP„¨¬šNþ bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP-YcNÿ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPª¿E‹J€ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPùÄSÌN bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÎóè¬N‚ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPx¦D)Nƒ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPEñÑH„ bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP” r‡@… b; particle/accuracy_4_of_7"trainer/global_step2:JP{˜aSN† bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP¶uÛJ‡ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¤­«UHˆ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP•˜ŸˆN‰ bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPSi{,HŠ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP£]ªæH‹ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPb¤÷;JŒ bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPZ”(ßN bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP­¹ö¢HŽ bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPòz 9J bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÑ·d)J bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPk€¹J‘ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPeÙJ’ bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¬ä[H“ bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPЛz?H” bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP¶/ÆêJ• bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPBN– bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPkºŸ›N— bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPoÔ¶WH˜ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP3ÂßÒJ™ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¢þ¬ôNš bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÑm*.N› bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPî+ Jœ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP_ÅŒ–H bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPŵxHž bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPõŸ¼HŸ bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP ‰{½N  bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP{Æ­ìN¡ bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP'n²;¢ b6 jet/accuracy_1_of_2"trainer/global_step2:JPWÉ#ö@£ b; particle/accuracy_3_of_4"trainer/global_step2:JPzí{«H¤ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP^›^+J¥ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‰dCH¦ bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPãaN§ bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP'­/H¨ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÏ|Ç>J© bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPò #@ª b; particle/accuracy_3_of_7"trainer/global_step2:JP•D¾“J« bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP0ÒŠH¬ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPü5ÍfJ­ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPúsH® bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP…3òH¯ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP¦ëü”J° bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP^½~,H± bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPüMæH² bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPŸkµ‘N³ bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPBï8J´ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPü ÷‹Jµ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPp³4PH¶ bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP±#ŸzN· bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPag6–H¸ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPD¼”¡J¹ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÆýš–Nº bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPQB+J» bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP ž:˜H¼ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPáH½ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPrÎÆH¾ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPø[³ÈJ¿ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPœþ–âNÀ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPVw4NÁ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPócÐN bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÎéêHà bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP‚Ò~ NÄ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP©™t„NÅ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPNoNÆ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPoèÜ”JÇ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPdﻉHÈ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPo€HÉ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP{ìÔVNÊ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPE9ûßNË bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPOÂZJÌ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPÍp$ NÍ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP™b¿NÎ bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP°u`3NÏ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPŠ<5+HÐ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPNê bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP±,7ŸNë bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP‚SòNì bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÛ¾V$Jí bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP2su+Jî bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPº2`nNï bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¸êªKNð bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPŽ‘`œJñ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP´4WO;ò b6 jet/accuracy_3_of_7"trainer/global_step2:JP'F[×Hó bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP1Xç£Nô bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP|­Nõ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP“z]ÕHö bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPcNŒH÷ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP?þÉCHø bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPa°b¸Jù bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPœ£ûNú bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPEHï³Nû bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP"·2oNü bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP“('QHý bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPò§ôôHþ bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPbÅ~tNÿ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP©KÂJ€ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPŽÍÚ-N bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPi&I‹J‚ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP žÝþJƒ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPHTEÌN„ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPýDQnN… bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPøKPN† bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPojŽN‡ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP8RŒžNˆ bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¤ÈsðN‰ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPø…:NŠ bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPü±ß…N‹ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPg¿CHŒ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÞ ;ÖN bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP}`’uNŽ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP÷Ií5J bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPûûÛJ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP™4¨šJ‘ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP=ÃÁÌH’ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP,‹ÉïJ“ bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPâÝcN” bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPùl÷J• bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP³"ÊN– bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPËÿ“H— bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPåH˜ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPy¨8YN™ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPQœ{$Hš bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÁºépJ› bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPFBJœ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÔ(ô"H bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP,¿=õHž bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPš¶ÖîHŸ bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP @0H  bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¾­TH¡ bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPèI² N¢ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP†j™H£ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPüÜ,H¤ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP{‹HMN¥ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP®á¼H¦ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP€8åÂH§ bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPûdÂ…H¨ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPl.®H© bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP™­ñJª bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP:äH« bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPyµX%J¬ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPoµ”J­ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPf­|J® bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP†€iN¯ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÕ—EJ° bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP S¦ËN± bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPEÎÚN² bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP3”ŸJ³ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP*W|2J´ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPéëŠ÷Hµ bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPzŠøLH¶ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPÈ1ÀôJ· bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP h‹2J¸ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP-„ÉN¹ bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPPÀ£@Jº bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPXºi¼N» bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP ÔZH¼ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP©`ŸH½ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP÷`N¾ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP“ýçyH¿ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÛœô+HÀ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPˆmékNÁ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPŠzñ; b6 jet/accuracy_3_of_3"trainer/global_step2:JPÜ”Nà bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPð>4JÄ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPí`b@Å b; particle/accuracy_2_of_7"trainer/global_step2:JPì:(JÆ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPPŒ[WNÇ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPŠÊhNÈ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¬âÈJÉ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP1 Œ¯HÊ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPÐÍTÂJË bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPÄà³(HÌ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP¿-ÌJÍ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¸âq|JÎ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPuÉjÚHÏ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPý25ŒJÐ bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP˜ó’NÑ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPžÚagJÒ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÊ. JÓ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPº˜Ù NÔ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÐí§HÕ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPHÜ-†JÖ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPƒ{N× bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP •IgJØ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÏÁ ;Ù b6 jet/accuracy_2_of_5"trainer/global_step2:JP=W@Ú b; particle/accuracy_1_of_4"trainer/global_step2:JPåáÀ;HÛ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPÿê0NÜ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP¤%NÝ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP÷ˆo-HÞ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPm‰VJß bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPŒ}Hà bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP…¢¶Ná bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP“rîJâ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‘PþŽHã bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP”•$Hä bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP. ¬èJå bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPINæ bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP]aŸJç bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÐàxhNè bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPþ<'SNé bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP×éJê bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPg7œ“Jë bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“À;ôNì bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPè±òHø bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP»0ÕHù bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPè‚ü¥Nú bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP1=ï©;û b6 jet/accuracy_2_of_3"trainer/global_step2:JP9ÃJü bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP'Ò›;ý b6 jet/accuracy_6_of_6"trainer/global_step2:JPo­ŸßJþ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP ([*Jÿ bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPüìJ€ bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP_ŽVbJ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP÷ÅùN‚ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP“ozNƒ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPUa—ÌN„ bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÚM‚N… bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPI ªN˜ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPï9A¶N™ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP,ÔBkNš bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP@@8k@› b; particle/accuracy_2_of_5"trainer/global_step2:JP¿Z¬­Jœ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPd{îDN bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP«å1AJž bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPq±JŸ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPLM;J  bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¸ªê N¡ bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPstzN¢ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP´QS3N£ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP"ÚBH¤ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP&¾šKJ¥ bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPØV‚ÙH¦ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP&ãÒ=H§ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPfE'H¨ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP\³ØN© bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP3øžNª bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPoH ÿJ« bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP$û¨éH¬ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPA”…@­ b; particle/accuracy_4_of_4"trainer/global_step2:JPQbxH® bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPëàt´N¯ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÉÞH° bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPYÃH± bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP«‰éH² bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP’ÓJ³ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŠ ÅýJ´ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPšíí§Jµ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP€&hN¶ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÇ%N· bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPže#W@¸ b; particle/accuracy_2_of_3"trainer/global_step2:JP'!åtJ¹ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPãUhiNº bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP‹geN» bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP) J¼ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP}9}k2½ - _wandbruntime‚1220  _runtime‚1220µÜÉ H ¾ ´¥ A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4510669708251953 particle/accuracy_2_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0015495087718591094 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8912530541419983 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.053030308336019516 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.20261436700820923 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9961450695991516 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0025133169256150723 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.24444441497325897 particle/accuracy_1_of_4‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.06666667014360428 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4287644922733307 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09827587008476257 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.09546571224927902 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.560606062412262 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1319444477558136 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.36784663796424866 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1785746067762375 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 particle/accuracy_1_of_3‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.007258064113557339 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.12971794605255127 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.08980782330036163 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.21450616419315338 6validation_average_jet_accuracy‚0.2992251217365265 particle/accuracy_5_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10350093990564346 8 Purity/1FRt*SRqqt2FBt©P§Lù/FRt_purity‚0.17937444150447845 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.13528648018836975 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 jet/accuracy_6_of_6‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.23036549985408783 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0680675208568573 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9927314519882202 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9772727489471436 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3113257884979248 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.019712550565600395 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3973214328289032 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10825793445110321 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1236080452799797 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.07619047164916992 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166865348816 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0076147448271512985 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09196603298187256 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11619096994400024 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.22583860158920288 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.06154537945985794 particle/accuracy_4_of_4‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.07894736528396606 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.17394062876701355 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4264639616012573 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_3‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.26293104887008667 particle/accuracy_4_of_5‚1 particle/accuracy_1_of_6‚1 jet/accuracy_1_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 jet/accuracy_2_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1105671152472496 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.0955132246017456 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.20217859745025635 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_5‚1 ,particle/sensitivity‚0.20797917246818542 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9774088263511658 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9704445004463196 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05758434161543846 particle/accuracy_3_of_6‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15366151928901672 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3224206268787384 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.17625266313552856 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.15538692474365234 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2126869112253189 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9900409579277039 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18521183729171753 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.08611111342906952 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3819444477558136 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21531778573989868 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004251700360327959 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1839604526758194 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 trainer/global_step‚1208 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9786860942840576 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9572878479957581 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.027073733508586884 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.16287879645824432 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08686868101358414 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1801244169473648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.15163691341876984 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 +jet/accuracy_3_of_4‚0.02638719230890274 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9919962882995605 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.003359242109581828 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_2_of_2‚0.054223377257585526 +jet/accuracy_2_of_6‚0.26923078298568726 jet/accuracy_5_of_7‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.7552585005760193 particle/accuracy_1_of_1‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30358877778053284 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.21365469694137573 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14022988080978394 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10629881173372269 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.125 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.14230702817440033 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06459219753742218 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.05882352963089943 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9913793206214905 9 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.047892723232507706 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1194869801402092 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.41345763206481934 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0011337868636474013 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0513506717979908 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10933472961187363 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9733134508132935 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29012349247932434 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4086287021636963 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08656852692365646 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.7850742340087891 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12093137204647064 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11417917907238007 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01785714365541935 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.21605876088142395 )loss/val_total_loss‚67.55873485999902 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6225349307060242 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37893080711364746 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10718416422605515 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07566434890031815 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15672512352466583 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11190474778413773 )jet/accuracy_1_of_3‚0.676074743270874 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23287619650363922 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.21132606267929077 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 particle/accuracy_2_of_6‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10031714290380478 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2424374222755432 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.03703703731298447 jet/accuracy_5_of_6‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.10450450330972672 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_4_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8994709849357605 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21581296622753143 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3998044729232788 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_1_of_1‚0.31950563192367554 particle/accuracy_5_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 % _timestamp‚1.7884292418413053e+09 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07047571986913681 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9955430626869202 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.285067617893219 particle/accuracy_2_of_2‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1041666716337204 particle/accuracy_6_of_6‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4180620014667511 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_7_of_7‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.07330247014760971 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9744922518730164 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9475839138031006 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2045019268989563 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8625731468200684 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.17861811816692352 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09604088962078094 *jet/accuracy_1_of_6‚0.8717949390411377 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2477818876504898 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16708719730377197 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2857142984867096 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9866589903831482 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_3‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_4_of_5‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9814498424530029 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9763166904449463 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07047571986913681 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.21565574407577515 +particle/specificity‚0.8674206733703613 particle/accuracy_1_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2320377379655838 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.0625 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1101190447807312 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.2149258702993393 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.15697024762630463 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.17633570730686188 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.23890459537506104 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9955430626869202 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.04707936942577362 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.16980208456516266 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.10833332687616348 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.15591029822826385 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_5_of_5‚0 (particle/accuracy‚0.7644417881965637 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1198803186416626 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.20155039429664612 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1518518626689911 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.09375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10486431419849396 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42077168822288513 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.003968254197388887 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_7_of_7‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9908630847930908 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9857456684112549 *jet/accuracy_2_of_4‚0.2265019565820694 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4165898561477661 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10085127502679825 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009006137028336525 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.15161237120628357 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.125 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.14519520103931427 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.2966666519641876 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.006519274320453405 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1090887114405632 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3196831941604614 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9788312911987305 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9652321338653564 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03058072365820408 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_2‚1 ,jet/accuracy_3_of_6‚0.057692307978868484 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.03842788189649582 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.057777781039476395 particle/accuracy_4_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.17864178121089935 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9916218519210815 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333335399627686 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07939401268959045 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9832764863967896 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.178068608045578 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9815776348114014 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9721575975418091 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9941798448562622 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5471777319908142 particle/accuracy_3_of_4‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1527777761220932 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.19091354310512543 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9876115918159485 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.24074074625968933 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3749212920665741 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.969804584980011 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9416563510894775 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 epoch‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6022792458534241 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.560606062412262 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03562945872545242 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9748067855834961 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9536458849906921 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13203474879264832 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.17719298601150513 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9956259727478027 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.37051284313201904 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9893125891685486 6 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.979651153087616 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12568147480487823 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1839604526758194 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0026954179629683495 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.0833333358168602 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06949648261070251 _step‚98 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.11067020148038864 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8023446798324585 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09719003736972809 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.18521720170974731 particle/accuracy_4_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 _runtime‚1220.198668966 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.14698922634124756 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.20316199958324432 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.49122804403305054 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.24574871361255646 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 jet/accuracy_3_of_7‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.18122199177742004 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.14021508395671844 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.11666665971279144 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.12129629403352737 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.06842576712369919 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3485054075717926 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.22974464297294617 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11069763451814651 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9832130074501038 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636494278907776 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8z UGz Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12733982503414154 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.17142857611179352 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.21271932125091553 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.058313317596912384 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01714421436190605 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4043583273887634 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9818809628486633 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9650435447692871 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.010736625641584396¦!*'xx¿ ¸¥ ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.057777781039476395 particle/accuracy_4_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.17864178121089935 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9916218519210815 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333335399627686 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07939401268959045 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9832764863967896 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.178068608045578 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9815776348114014 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9721575975418091 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9941798448562622 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5471777319908142 particle/accuracy_3_of_4‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1527777761220932 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.19091354310512543 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9876115918159485 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.24074074625968933 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3749212920665741 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.969804584980011 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9416563510894775 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 epoch‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6022792458534241 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.560606062412262 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03562945872545242 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9748067855834961 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9536458849906921 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13203474879264832 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.17719298601150513 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9956259727478027 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.37051284313201904 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9893125891685486 6 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.979651153087616 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12568147480487823 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1839604526758194 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0026954179629683495 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.0833333358168602 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06949648261070251 _step‚98 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.11067020148038864 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8023446798324585 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09719003736972809 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.18521720170974731 particle/accuracy_4_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 _runtime‚1220.198668966 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.14698922634124756 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.20316199958324432 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.49122804403305054 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.24574871361255646 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 jet/accuracy_3_of_7‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.18122199177742004 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.14021508395671844 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.11666665971279144 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.12129629403352737 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.06842576712369919 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3485054075717926 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.22974464297294617 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11069763451814651 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9832130074501038 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636494278907776 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12733982503414154 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.17142857611179352 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.21271932125091553 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.058313317596912384 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01714421436190605 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4043583273887634 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9818809628486633 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9650435447692871 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.010736625641584396 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4510669708251953 particle/accuracy_2_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0015495087718591094 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8912530541419983 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.053030308336019516 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.20261436700820923 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9961450695991516 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0025133169256150723 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.24444441497325897 particle/accuracy_1_of_4‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.06666667014360428 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4287644922733307 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09827587008476257 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.09546571224927902 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.560606062412262 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1319444477558136 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.36784663796424866 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1785746067762375 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 particle/accuracy_1_of_3‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.007258064113557339 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.12971794605255127 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.08980782330036163 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.21450616419315338 6validation_average_jet_accuracy‚0.2992251217365265 particle/accuracy_5_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10350093990564346 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.17937444150447845 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.13528648018836975 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 jet/accuracy_6_of_6‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.23036549985408783 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0680675208568573 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9927314519882202 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9772727489471436 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3113257884979248 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.019712550565600395 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3973214328289032 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10825793445110321 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1236080452799797 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.07619047164916992 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166865348816 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0076147448271512985 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09196603298187256 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11619096994400024 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.22583860158920288 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.06154537945985794 particle/accuracy_4_of_4‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.07894736528396606 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.17394062876701355 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4264639616012573 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_3‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.26293104887008667 particle/accuracy_4_of_5‚1 particle/accuracy_1_of_6‚1 jet/accuracy_1_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 jet/accuracy_2_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1105671152472496 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.0955132246017456 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.20217859745025635 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_5‚1 ,particle/sensitivity‚0.20797917246818542 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9774088263511658 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9704445004463196 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05758434161543846 particle/accuracy_3_of_6‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15366151928901672 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3224206268787384 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.17625266313552856 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.15538692474365234 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2126869112253189 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9900409579277039 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18521183729171753 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.08611111342906952 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3819444477558136 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21531778573989868 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004251700360327959 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1839604526758194 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 trainer/global_step‚1208 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9786860942840576 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9572878479957581 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.027073733508586884 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.16287879645824432 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08686868101358414 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1801244169473648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.15163691341876984 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 +jet/accuracy_3_of_4‚0.02638719230890274 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9919962882995605 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.003359242109581828 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_2_of_2‚0.054223377257585526 +jet/accuracy_2_of_6‚0.26923078298568726 jet/accuracy_5_of_7‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.7552585005760193 particle/accuracy_1_of_1‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30358877778053284 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.21365469694137573 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14022988080978394 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10629881173372269 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.125 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.14230702817440033 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06459219753742218 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.05882352963089943 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9913793206214905 9 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.047892723232507706 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1194869801402092 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.41345763206481934 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0011337868636474013 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0513506717979908 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10933472961187363 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9733134508132935 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29012349247932434 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4086287021636963 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08656852692365646 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.7850742340087891 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12093137204647064 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11417917907238007 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01785714365541935 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.21605876088142395 )loss/val_total_loss‚67.55873485999902 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6225349307060242 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37893080711364746 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10718416422605515 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07566434890031815 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15672512352466583 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11190474778413773 )jet/accuracy_1_of_3‚0.676074743270874 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23287619650363922 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.21132606267929077 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 particle/accuracy_2_of_6‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10031714290380478 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2424374222755432 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.03703703731298447 jet/accuracy_5_of_6‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.10450450330972672 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_4_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8994709849357605 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21581296622753143 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3998044729232788 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_1_of_1‚0.31950563192367554 particle/accuracy_5_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 % _timestamp‚1.7884292418413053e+09 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07047571986913681 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9955430626869202 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.285067617893219 particle/accuracy_2_of_2‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1041666716337204 particle/accuracy_6_of_6‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4180620014667511 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_7_of_7‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.07330247014760971 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9744922518730164 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9475839138031006 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2045019268989563 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8625731468200684 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.17861811816692352 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09604088962078094 *jet/accuracy_1_of_6‚0.8717949390411377 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2477818876504898 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16708719730377197 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2857142984867096 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9866589903831482 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_3‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracyÕê‚G_4_of_5‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9814498424530029 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9763166904449463 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07047571986913681 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.21565574407577515 +particle/specificity‚0.8674206733703613 particle/accuracy_1_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2320377379655838 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.0625 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1101190447807312 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.2149258702993393 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.15697024762630463 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.17633570730686188 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.23890459537506104 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9955430626869202 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.04707936942577362 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.16980208456516266 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.10833332687616348 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.15591029822826385 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_5_of_5‚0 (particle/accuracy‚0.7644417881965637 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1198803186416626 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.20155039429664612 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1518518626689911 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.09375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10486431419849396 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42077168822288513 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.003968254197388887 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_7_of_7‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9908630847930908 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9857456684112549 *jet/accuracy_2_of_4‚0.2265019565820694 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4165898561477661 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10085127502679825 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009006137028336525 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.15161237120628357 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.125 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.14519520103931427 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.2966666519641876 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.006519274320453405 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1090887114405632 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3196831941604614 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9788312911987305 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9652321338653564 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03058072365820408 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_2‚1 ,jet/accuracy_3_of_6‚0.057692307978868484 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.03842788189649582bò­ÆÔ’À ‚(:‡ ÇåÔˆ‹‚· gpu.0.gpu‚12.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pZUGÁ ‚(:¼ ÇåÔ³—¹ disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚5782.54345703125" disk.dm-0.out‚5782.54345703125 disk.drbd1716.in‚0# proc.memory.availableMB‚2600.25* proc.memory.percent‚29.973649978637695 network.recv‚488641 disk.sda1.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚4.078125 disk.sda.out‚ 44.99609375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 44.99609375 network.sent‚ 1.702412e+06 disk.sda.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0$ memory_percent‚84.12933349609375 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.078125 disk.drbd2080.in‚0 cpu‚25.305860589311823) disk./.usagePercent‚17.99973279981636$ disk./.usageGB‚268.2090263366699% disk.nvme0n1.out‚5782.54345703125 disk.dm-1.out‚ 44.99609375 disk.rbd1.out‚ 0.61328125" proc.memory.rssMB‚ 4910.8828125ú68ž ‚(:“ Ò‡åÔ”½þ¶ gpu.0.gpu‚63.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚285.002' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.000400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÖ¯¸Ià ‚(:¾ Ò‡åÔ¡‘ø¸ disk.sda.in‚0 disk.sda.out‚ 45.64453125 disk.nvme0n1.in‚4.078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 cpu‚25.29163605079104) disk./.usagePercent‚18.14717931513192 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚4.078125' disk.nvme0c0n1.out‚10521.5224609375 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚4.078125 network.sent‚ 1.826145e+06 network.recv‚494868% disk.nvme0n1.out‚10521.5224609375 disk.rbd1.out‚ 0.61328125 disk.drbd2080.in‚0' proc.memory.availableMB‚ 4479.390625$ disk./.usageGB‚270.4060859680176 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 45.64453125$ memory_percent‚72.65996932983398 disk.dm-1.out‚ 45.64453125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚18.526744842529297 disk.drbd2266.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚10521.5224609375 disk.sda1.out‚0! proc.memory.rssMB‚ 3035.421875 proc.cpu.threads‚17 disk.drbd2265.in‚0s^‘Ä ‚(:† á‡åÔžü‚µ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚177.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RûÁîKÅ ‚(:À á‡åÔ˜„Á¶ disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 network.sent‚ 1.83233e+06 disk.sda3.out‚ 46.30078125 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0% disk.nvme0n1.out‚10547.7490234375! proc.memory.rssMB‚ 3035.421875 cpu‚25.288916915538607 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-0.in‚4.078125 disk.sda2.in‚0 disk.dm-1.in‚0$ memory_percent‚72.66461849212646 network.recv‚499666 disk.sda.out‚ 46.30078125 disk.nvme0n1.in‚4.078125 disk.sda2.out‚0' disk.nvme0c0n1.out‚10547.7490234375" disk.dm-0.out‚10547.7490234375* proc.memory.percent‚18.526744842529297 disk.drbd2265.out‚0 disk.dm-1.out‚ 46.30078125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125) proc.memory.availableMB‚ 4478.62890625* disk./.usagePercent‚18.147767620445606# disk./.usageGB‚270.414852142334 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0VBê2Æ - _wandbruntime‚1271  _runtime‚1271Àx‰ÈÇ Â ' lr-AdamW/pg1‚0.00020456360541286874 ' lr-AdamW/pg2‚0.00020456360541286874 trainer/global_step‚1249 % _timestamp‚1.7884292935390875e+09 _runtime‚1271.894125839 _step‚99a—FÚÌÈ Æ % _timestamp‚1.7884292935390875e+09 _runtime‚1271.894125839 _step‚99 ' lr-AdamW/pg1‚0.00020456360541286874 ' lr-AdamW/pg2‚0.00020456360541286874 trainer/global_step‚1249c³îg¤2É - _wandbruntime‚1271  _runtime‚1271ʦ´ø'Ê òO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26947865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13429115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22445591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26924761 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167138576438272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22445585 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49393456 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2668837165303 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145314481755648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13429112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170053422704640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3717368966656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172723677177856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170053449096192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9817917 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 trainer/global_step‚1249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26924761 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149031931774976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145314509890048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26947865 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3717421884928 % _timestamp‚1.7884292935398211e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172723677177856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170053449096192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2670254473216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172726808962048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170053422704640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2665733696387 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456•žü'Ë öO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172726808962048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170053422704640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2665733696387 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26947865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13429115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22445591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26924761 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167138576438272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22445585 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49393456 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2668837165303 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145314481755648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13429112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170053422704640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3717368966656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172723677177856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170053449096192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9817917 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 trainer/global_step‚1249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26924761 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149031931774976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145314509890048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26947865 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3717421884928 % _timestamp‚1.7884292935398211e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172723677177856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170053449096192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2670254473216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984d_klM2Ì - _wandbruntime‚1272  _runtime‚1272Zü*—–Í úM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26968981 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 N[T¼j%ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26945851 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172857283579904 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23265101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 # _timestamp‚1.78842929444093e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 trainer/global_step‚1249 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78942208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2670945958611 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49408720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22462875 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2672366787072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26968981 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149147292593664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167267998867456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2672366787072 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22462875 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149147213651456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50807808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49431856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145426920754176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3720321031680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170184916792832 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13439365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49431856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170184943184384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22462869 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145426892619776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50807808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49431856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145426920754176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3720321031680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170184916792832 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13439365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49431856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170184943184384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22462869 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145426892619776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26968981 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26945851 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172857283579904 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23265101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 # _timestamp‚1.78842929444093e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 trainer/global_step‚1249 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78942208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2670945958611 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49408720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22462875 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2672366787072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26968981 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149147292593664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167267998867456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2672366787072 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22462875 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149147213651456e+õ2Ï - _wandbruntime‚1272  _runtime‚1272±%šXÐ Ò 2loss/SRqqt4/assignment_loss‚2.9279354168818545 /loss/FBt4/assignment_loss‚0.469628156336156 /loss/FRt1/detection_loss‚2.0337198007674444 /loss/FRt2/detection_loss‚2.6929274717966716 0loss/SRqqt4/detection_loss‚2.815734008183846 .loss/FBt1/detection_loss‚1.956218905172291 /loss/FRt1/assignment_loss‚3.813199955319601 1loss/FBt3/assignment_loss‚0.12296124892114808 /loss/FRt4/detection_loss‚3.3201056096864785 0loss/FRt2/assignment_loss‚5.3857096592585245 epoch‚1 trainer/global_step‚1249 2loss/SRqqt1/assignment_loss‚1.9679736101999878 0loss/FBt1/assignment_loss‚0.4250497829591012 0loss/FBt2/assignment_loss‚0.6102389292791486 1loss/SRqqt1/detection_loss‚1.5645531053344408 /loss/FBt3/detection_loss‚3.1047155583898225 2loss/SRqqt3/assignment_loss‚1.3682466836865335 &loss/total_loss‚60.247371786455574 % _timestamp‚1.7884292944430282e+09 _runtime‚1272.797995071 _step‚102 /loss/FRt4/assignment_loss‚5.784451194431471 /loss/FRt3/detection_loss‚2.2872950481043923 1loss/SRqqt3/detection_loss‚1.7125442242797684 .loss/FBt2/detection_loss‚4.212604522705078 .loss/FBt4/detection_loss‚2.307598115758844 /loss/FRt3/assignment_loss‚4.148478359646267 1loss/SRqqt2/assignment_loss‚2.413068825379014 1loss/SRqqt2/detection_loss‚2.8024135939776897€@°7\Ñ Ö % _timestamp‚1.7884292944430282e+09 _runtime‚1272.797995071 _step‚102 /loss/FRt4/assignment_loss‚5.784451194431471 /loss/FRt3/detection_loss‚2.2872950481043923 1loss/SRqqt3/detection_loss‚1.7125442242797684 .loss/FBt2/detection_loss‚4.212604522705078 .loss/FBt4/detection_loss‚2.307598115758844 /loss/FRt3/assignment_loss‚4.148478359646267 1loss/SRqqt2/assignment_loss‚2.413068825379014 1loss/SRqqt2/detection_loss‚2.8024135939776897 2loss/SRqqt4/assignment_loss‚2.9279354168818545 /loss/FBt4/assignment_loss‚0.469628156336156 /loss/FRt1/detection_loss‚2.0337198007674444 /loss/FRt2/detection_loss‚2.6929274717966716 0loss/SRqqt4/detection_loss‚2.815734008183846 .loss/FBt1/detection_loss‚1.956218905172291 /loss/FRt1/assignment_loss‚3.813199955319601 1loss/FBt3/assignment_loss‚0.12296124892114808 /loss/FRt4/detection_loss‚3.3201056096864785 0loss/FRt2/assignment_loss‚5.3857096592585245 epoch‚1 trainer/global_step‚1249 2loss/SRqqt1/assignment_loss‚1.9679736101999878 0loss/FBt1/assignment_loss‚0.4250497829591012 0loss/FBt2/assignment_loss‚0.6102389292791486 1loss/SRqqt1/detection_loss‚1.5645531053344408 /loss/FBt3/detection_loss‚3.1047155583898225 2loss/SRqqt3/assignment_loss‚1.3682466836865335 &loss/total_loss‚60.247371786455574f‚OÒ ‚(:’ ð‡åԢŹ¶ gpu.0.gpu‚47.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚151.26' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.251999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gòì™NÓ ‚(:à ð‡åԊ騸 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚503189 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125$ disk./.usageGB‚270.4238815307617( disk.nvme0c0n1.out‚10598.05712890625 disk.rbd1.in‚0$ memory_percent‚72.67518043518066* proc.memory.percent‚18.526744842529297 disk.sda3.out‚ 46.63671875 disk.drbd2080.out‚0 disk.dm-1.out‚ 46.63671875 cpu‚25.28628076303129 disk.nvme0c0n1.in‚4.078125 disk.drbd2080.in‚0 disk.sda.out‚ 46.63671875 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0* disk./.usagePercent‚18.148373590279153 disk.dm-0.in‚4.078125# disk.dm-0.out‚10598.05712890625 disk.sda1.in‚0 disk.drbd2265.in‚0 network.sent‚ 1.837678e+06 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.078125 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda.in‚0& disk.nvme0n1.out‚10598.05712890625( proc.memory.availableMB‚ 4476.8984375! proc.memory.rssMB‚ 3035.421875 disk.rbd0.in‚0.,  žÔ ‚(:“ ÿ‡åԟשµ gpu.0.gpu‚67.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.462199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÍÿ 7Õ ‚(:¬ ÿ‡åÔػ϶& proc.memory.availableMB‚ 4476.53125! proc.memory.rssMB‚ 3035.421875 disk.rbd0.in‚0 disk.rbd1.out‚ 0.76171875 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17% disk./.usageGB‚270.42773818969727 disk.nvme0n1.in‚4.078125 disk.dm-1.in‚0# disk.dm-0.out‚10635.37744140625 network.sent‚ 1.945345e+06 network.recv‚509822 disk.sda2.out‚0 disk.dm-1.out‚47.5 disk.sda3.out‚47.5 disk.drbd2080.in‚0 disk.sda.out‚47.5 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚10635.37744140625$ memory_percent‚72.67742156982422 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0* proc.memory.percent‚18.526744842529297) disk./.usagePercent‚18.14863241389627 disk.sda.in‚0 cpu‚25.28346477489482& disk.nvme0n1.out‚10635.37744140625 disk.dm-0.in‚4.078125 disk.drbd1716.in‚0K¤×˜Ö ‚(:’ ŽˆåÔôȲ· gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚300.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.08859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ÔÒðF× ‚(:» ŽˆåԢݹ disk.rbd1.in‚0 disk.sda.out‚48.15625 disk.nvme0c0n1.in‚4.078125 disk.sda3.out‚48.15625& disk.nvme0n1.out‚10674.75537109375 network.sent‚ 1.951291e+06% disk./.usageGB‚270.43132400512695 disk.sda.in‚0) proc.memory.availableMB‚ 4476.76953125! proc.memory.rssMB‚ 3035.421875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚10674.75537109375 disk.rbd0.in‚0 disk.dm-1.out‚48.15625* proc.memory.percent‚18.526744842529297 disk.drbd2266.in‚0 disk.dm-0.in‚4.078125 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0# memory_percent‚72.6759672164917* disk./.usagePercent‚18.148873060978453 disk.rbd1.out‚ 0.76171875# disk.dm-0.out‚10674.75537109375 disk.rbd0.out‚0 cpu‚25.28054591387425 disk.nvme0n1.in‚4.078125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚513727 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0êï –2Ø - _wandbruntime‚1319  _runtime‚1319ïAjÇÙ Á & lr-AdamW/pg2‚0.0002045358958881695 trainer/global_step‚1299 % _timestamp‚1.7884293407887123e+09 _runtime‚1319.143727225 _step‚103 & lr-AdamW/pg1‚0.0002045358958881695tNíËÚ Å & lr-AdamW/pg2‚0.0002045358958881695 trainer/global_step‚1299 % _timestamp‚1.7884293407887123e+09 _runtime‚1319.143727225 _step‚103 & lr-AdamW/pg1‚0.0002045358958881695gž˜”Ü2Û - _wandbruntime‚1319  _runtime‚1319QÐ$‰£Ü ôO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179404011694080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179404011694080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23309785 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81052672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 trainer/global_step‚1299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13941765 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176628127114240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176382742338551 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52918272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51292398 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28005717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3865405653504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176379613441667 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2778989972480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150935024962048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173608451927040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173608426499072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23309785 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3865352735232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176628153505792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13941762 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2774290411511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27982613 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154800377697280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27982613 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2775884579840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51315508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 W@øþ'Ý øO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27982613 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2775884579840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51315508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179404011694080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179404011694080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23309785 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81052672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 trainer/global_step‚1299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13941765 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176628127114240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176382742338551 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52918272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51292398 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28005717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3865405653504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176379613441667 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2778989972480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150935024962048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173608451927040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173608426499072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23309785 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3865352735232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176628153505792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13941762 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2774290411511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27982613 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154800377697280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0hB¾zÊ“Þ ‚(:ˆ ˆåÔóå³µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚174.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÒÎ$Qß ‚(:Æ ˆåÔ™»·( disk.nvme0c0n1.out‚10711.67333984375 cpu‚25.27805472213333 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda3.out‚ 48.85546875 disk.sda1.out‚0) proc.memory.availableMB‚ 4476.29296875! proc.memory.rssMB‚ 3035.421875# disk.dm-0.out‚10711.67333984375 network.sent‚ 1.957431e+06 disk.sda.out‚ 48.85546875 disk.nvme0n1.in‚4.078125 disk.rbd1.out‚ 0.7939453125* proc.memory.percent‚18.526744842529297* disk./.usagePercent‚18.149596538270217 disk.rbd0.out‚0 disk.dm-1.out‚ 48.85546875 disk.dm-0.in‚4.078125 disk.drbd1716.in‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 network.recv‚518348 disk.drbd2265.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚10711.67333984375 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚72.67887592315674 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.078125$ disk./.usageGB‚270.4421043395996 disk.drbd1716.out‚0 disk.rbd1.in‚0Bd”2à - _wandbruntime‚1320  _runtime‚1320R ='á þM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24161438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 trainer/global_step‚1299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51353908 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2777996893696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179540749893632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151047435826176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51330772 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 _step‚105 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176759621202432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23327075 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23327069 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28003703 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78942208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13952015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10208941 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51353908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13952018 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176759647593984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176759647593984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154915758304256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781102299648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154915837246464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51330772 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24160956 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28026833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3868373285888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3868322478080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884293416950371e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176514248133075 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176511119235983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173737823500288 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10209420’£†>`â ‚N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 _step‚105 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176759621202432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23327075 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23327069 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28003703 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78942208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13952015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10208941 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51353908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13952018 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176759647593984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176759647593984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154915758304256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781102299648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154915837246464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51330772 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24160956 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28026833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3868373285888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3868322478080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884293416950371e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176514248133075 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.laUd«¨rge_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176511119235983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173737823500288 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10209420 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24161438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 trainer/global_step‚1299 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51353908 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2777996893696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179540749893632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151047435826176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51330772i6‘Þ2ã - _wandbruntime‚1320  _runtime‚1320M,&‚Wä Ñ 1loss/SRqqt1/detection_loss‚1.7340794360408416 .loss/FBt1/detection_loss‚2.308466015383601 /loss/FBt4/detection_loss‚1.9033639919842245 /loss/FRt4/assignment_loss‚4.934203954843374 1loss/SRqqt2/detection_loss‚3.3894071096465703 /loss/FBt3/detection_loss‚3.6432681560516356 trainer/global_step‚1299 % _timestamp‚1.7884293416971834e+09 /loss/FBt1/assignment_loss‚0.331384577780418 /loss/FBt2/detection_loss‚3.7174470300475755 %loss/total_loss‚61.69096449750926 epoch‚1 2loss/SRqqt2/assignment_loss‚2.7794195072991505 0loss/FBt3/assignment_loss‚0.1259767546580406 /loss/FRt3/detection_loss‚2.1157365229093665 1loss/SRqqt3/detection_loss‚1.4699102735672243 /loss/FRt2/assignment_loss‚6.167289241667716 2loss/SRqqt3/assignment_loss‚1.8098152164274302 /loss/SRqqt4/detection_loss‚2.85916327346455 0loss/FRt1/assignment_loss‚3.8840848151242957 0loss/FBt4/assignment_loss‚0.3686460134932706 _runtime‚1320.052124824 _step‚106 2loss/SRqqt1/assignment_loss‚2.0437853949574323 2loss/SRqqt4/assignment_loss‚2.7335447560657156 0loss/FBt2/assignment_loss‚0.5658604985207294 .loss/FRt1/detection_loss‚2.157448336763202 .loss/FRt4/detection_loss‚4.175776775066669 0loss/FRt3/assignment_loss‚3.8291698874849263 /loss/FRt2/detection_loss‚2.64371695826130540ÂÏú[å Õ /loss/FRt3/detection_loss‚2.1157365229093665 1loss/SRqqt3/detection_loss‚1.4699102735672243 /loss/FRt2/assignment_loss‚6.167289241667716 2loss/SRqqt3/assignment_loss‚1.8098152164274302 /loss/SRqqt4/detection_loss‚2.85916327346455 0loss/FRt1/assignment_loss‚3.8840848151242957 0loss/FBt4/assignment_loss‚0.3686460134932706 _runtime‚1320.052124824 _step‚106 2loss/SRqqt1/assignment_loss‚2.0437853949574323 2loss/SRqqt4/assignment_loss‚2.7335447560657156 0loss/FBt2/assignment_loss‚0.5658604985207294 .loss/FRt1/detection_loss‚2.157448336763202 .loss/FRt4/detection_loss‚4.175776775066669 0loss/FRt3/assignment_loss‚3.8291698874849263 /loss/FRt2/detection_loss‚2.6437169582613054 1loss/SRqqt1/detection_loss‚1.7340794360408416 .loss/FBt1/detection_loss‚2.308466015383601 /loss/FBt4/detection_loss‚1.9033639919842245 /loss/FRt4/assignment_loss‚4.934203954843374 1loss/SRqqt2/detection_loss‚3.3894071096465703 /loss/FBt3/detection_loss‚3.6432681560516356 trainer/global_step‚1299 % _timestamp‚1.7884293416971834e+09 /loss/FBt1/assignment_loss‚0.331384577780418 /loss/FBt2/detection_loss‚3.7174470300475755 %loss/total_loss‚61.69096449750926 epoch‚1 2loss/SRqqt2/assignment_loss‚2.7794195072991505 0loss/FBt3/assignment_loss‚0.1259767546580406jXÚTžæ ‚(:“ ¬ˆåÔìà±µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚303.289' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.657799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xí>OLç ‚(:Á ¬ˆåÔŒ¸¾· disk.dm-1.in‚0 disk.sda3.out‚ 49.30859375! proc.memory.rssMB‚ 3035.421875 cpu‚25.275526887525274 network.recv‚523889 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚10761.017578125 disk.dm-0.in‚4.078125 disk.drbd2265.in‚0* disk./.usagePercent‚18.150122377745546 disk.nvme0n1.in‚4.078125$ disk.nvme0n1.out‚10761.017578125 disk.drbd1716.in‚0$ disk./.usageGB‚270.4499397277832 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.rbd0.in‚0$ memory_percent‚72.67911434173584) proc.memory.availableMB‚ 4476.25390625 disk.rbd1.out‚ 0.7939453125! disk.dm-0.out‚10761.017578125 disk.drbd2265.out‚0 disk.rbd0.out‚0* proc.memory.percent‚18.526744842529297 proc.cpu.threads‚17 disk.dm-1.out‚ 49.30859375 disk.nvme0c0n1.in‚4.078125 disk.sda3.in‚0 disk.drbd2080.out‚0 network.sent‚ 2.054201e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda.out‚ 49.30859375ÆàLÚžè ‚(:“ »ˆåÔúŒ§µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚252.982' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.596399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;˜mQIé ‚(:¾ »ˆåÔÞë‹· disk.sda.out‚ 49.88671875 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚10801.482421875 disk.drbd2080.out‚0* proc.memory.percent‚18.526744842529297 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚10801.482421875& proc.memory.availableMB‚ 4475.78125 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 49.88671875 disk.rbd1.out‚ 0.85693359375 network.sent‚ 2.152077e+06) disk./.usagePercent‚18.15095261017909 disk.nvme0n1.in‚4.078125 disk.drbd2080.in‚0 disk.drbd2265.out‚0$ disk.nvme0n1.out‚10801.482421875 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 network.recv‚530344 disk.sda.in‚0 disk.nvme0c0n1.in‚4.078125 disk.dm-0.in‚4.078125 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 49.88671875 disk.sda1.in‚0 cpu‚25.272901052258774$ disk./.usageGB‚270.4623107910156 disk.sda1.out‚0 disk.drbd1716.out‚0$ memory_percent‚72.68199920654297! proc.memory.rssMB‚ 3035.421875}äžê ‚(:“ ʈåÔ©Ø°¶ gpu.0.gpu‚71.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚267.508' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.501599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WdçMë ‚(: ʈåԼЦ· disk.drbd2266.out‚0& disk.nvme0c0n1.out‚10843.505859375 disk.sda1.out‚0) proc.memory.availableMB‚ 4476.01953125 disk.drbd1716.in‚0 network.sent‚ 2.160103e+06 network.recv‚534366* disk./.usagePercent‚18.151163560387303 disk.nvme0n1.in‚4.078125 disk.sda2.out‚0 disk.rbd1.out‚ 0.85693359375 disk.dm-0.in‚4.078125 disk.drbd2266.in‚0 disk.sda3.in‚0$ memory_percent‚72.68054485321045 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.078125! disk.dm-0.out‚10843.505859375 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0* proc.memory.percent‚18.526744842529297 disk.dm-1.out‚ 50.41796875 disk.drbd2265.in‚0$ disk.nvme0n1.out‚10843.505859375$ disk./.usageGB‚270.4654541015625 disk.rbd0.in‚0 disk.sda3.out‚ 50.41796875 disk.sda1.in‚0 disk.sda.out‚ 50.41796875 cpu‚25.270489040815242 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0! proc.memory.rssMB‚ 3035.421875ŒFÉ~2ì - _wandbruntime‚1366  _runtime‚1366"ÑuÉí à ' lr-AdamW/pg1‚0.00020448687747447612 ' lr-AdamW/pg2‚0.00020448687747447612 trainer/global_step‚1349 % _timestamp‚1.7884293877964888e+09 _runtime‚1366.151580674 _step‚107ýJ= Íî Ç ' lr-AdamW/pg2‚0.00020448687747447612 trainer/global_step‚1349 % _timestamp‚1.7884293877964888e+09 _runtime‚1366.151580674 _step‚107 ' lr-AdamW/pg1‚0.00020448687747447612kø1P42ï - _wandbruntime‚1366  _runtime‚1366r‚ó"ù'ð óO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14454415 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180078276559872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2884620079104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24173985 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29063569 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24173991 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160568470194688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚186087477994496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160568553344512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182958045645559 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14454412 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183202831523840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2884620079104 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _runtime‚1366.152620326 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 $ _timestamp‚1.788429387797257e+09 _step‚108 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4012902026240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186084346210304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183202857915392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186087477994496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24173985 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29063569 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24173991 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160568470194688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚186087477994496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160568553344512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182958045645559 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14454412 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183202831523840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2884620079104 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _runtime‚1366.152620326 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 $ _timestamp‚1.788429387797257e+09 _step‚108 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4012902026240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186084346210304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183202857915392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186087477994496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25051953 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1367.037868281 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183334325612032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14464668 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180207698989056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24191275 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183334352003584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788429388682217e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚186221084409856 trainer/global_step‚1349 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53275960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14464665 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24191275 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156668007166976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25052457 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚186217952612352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2881852451027 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53252824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2886732406272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53275960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10587288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚186221084409856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10587789 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2883627000320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2878748981903 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29084685 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24191269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29084685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÕÓØÅ'ô N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚186221084409856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10587789 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2883627000320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2878748981903 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29084685 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24191269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29084685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24191269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2883627000320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29061555 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25051953 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1367.037868281 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183334325612032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14464668 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180207698989056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24191275 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183334352003584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788429388682217e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚186221084409856 trainer/global_step‚1349 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53275960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14464665 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24191275 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156668007166976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25052457 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚186217952612352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2881852451027 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53252824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2886732406272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53275960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10587288m,Ó©Ú2õ - _wandbruntime‚1367  _runtime‚1367:NbbWö Ñ % _timestamp‚1.7884293886840048e+09 .loss/FRt3/detection_loss‚2.094732690354188 1loss/SRqqt4/detection_loss‚2.6660938218787864 .loss/FBt1/detection_loss‚1.734402592536252 epoch‚1 2loss/SRqqt1/assignment_loss‚1.8870173062597002 1loss/SRqqt3/detection_loss‚1.3682142568840867 .loss/FBt2/detection_loss‚7.065792679786682 0loss/SRqqt1/detection_loss‚1.619309813422816 /loss/FRt1/assignment_loss‚3.708639394669306 2loss/SRqqt4/assignment_loss‚2.5800423202691256 1loss/FBt3/assignment_loss‚0.12529386692461808 /loss/FBt3/detection_loss‚2.9799725928089837 /loss/FRt2/assignment_loss‚5.924609151379816 /loss/FRt3/assignment_loss‚3.503162306547165 1loss/FBt1/assignment_loss‚0.43815571167785067 0loss/FBt2/assignment_loss‚0.9599692821502686 .loss/FRt1/detection_loss‚2.010169594060807 /loss/FRt4/assignment_loss‚6.149447524029275 1loss/FBt4/assignment_loss‚0.49765276877769704 .loss/FRt4/detection_loss‚3.108287593592768 /loss/FBt4/detection_loss‚2.6882730886773736 _runtime‚1367.038959903 2loss/SRqqt2/assignment_loss‚2.4088766620709348 1loss/SRqqt3/assignment_loss‚1.597735961366977 /loss/FRt2/detection_loss‚3.1267468908737444 0loss/SRqqt2/detection_loss‚2.951477344219501 trainer/global_step‚1349 _step‚110 &loss/total_loss‚63.194075215218724‘![÷ Õ /loss/FRt2/assignment_loss‚5.924609151379816 /loss/FRt3/assignment_loss‚3.503162306547165 1loss/FBt1/assignment_loss‚0.43815571167785067 0loss/FBt2/assignment_loss‚0.9599692821502686 .loss/FRt1/detection_loss‚2.010169594060807 /loss/FRt4/assignment_loss‚6.149447524029275 1loss/FBt4/assignment_loss‚0.49765276877769704 .loss/FRt4/detection_loss‚3.108287593592768 /loss/FBt4/detection_loss‚2.6882730886773736 _runtime‚1367.038959903 2loss/SRqqt2/assignment_loss‚2.4088766620709348 1loss/SRqqt3/assignment_loss‚1.597735961366977 /loss/FRt2/detection_loss‚3.1267468908737444 0loss/SRqqt2/detection_loss‚2.951477344219501 trainer/global_step‚1349 _step‚110 &loss/total_loss‚63.194075215218724 % _timestamp‚1.7884293886840048e+09 .loss/FRt3/detection_loss‚2.094732690354188 1loss/SRqqt4/detection_loss‚2.6660938218787864 .loss/FBt1/detection_loss‚1.734402592536252 epoch‚1 2loss/SRqqt1/assignment_loss‚1.8870173062597002 1loss/SRqqt3/detection_loss‚1.3682142568840867 .loss/FBt2/detection_loss‚7.065792679786682 0loss/SRqqt1/detection_loss‚1.619309813422816 /loss/FRt1/assignment_loss‚3.708639394669306 2loss/SRqqt4/assignment_loss‚2.5800423202691256 1loss/FBt3/assignment_loss‚0.12529386692461808 /loss/FBt3/detection_loss‚2.9799725928089837n?#®âø ‚(:’ ÙˆåÔý¦«¶ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.062' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.01240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»„,Rù ‚(:Ç ÙˆåÔŘû¶ disk.sda3.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚10884.81396484375$ memory_percent‚72.68269062042236 network.sent‚ 2.164178e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 cpu‚25.268149249204072 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 51.05859375 disk.drbd2080.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.078125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.078125 disk.dm-1.out‚ 51.05859375 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚10884.81396484375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.078125& disk.nvme0n1.out‚10884.81396484375) proc.memory.availableMB‚ 4475.66796875 network.recv‚539161* disk./.usagePercent‚18.152550609207818 disk.rbd1.in‚0 disk.sda.out‚ 51.05859375! proc.memory.rssMB‚ 3035.421875* proc.memory.percent‚18.526744842529297% disk./.usageGB‚270.48612213134766 disk.rbd1.out‚ 0.919921875†:?“ú ‚(:ˆ èˆåÔž¬Œµ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.626' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c‰‹ðIû ‚(:¾ èˆåÔ¢éî¶ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 network.recv‚544610 disk.drbd2080.in‚0# disk.nvme0n1.out‚10928.94921875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.26567356278508% disk./.usageGB‚270.49298095703125 disk.dm-1.out‚ 51.44921875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚4.078125 disk.drbd1716.in‚0 disk.sda.out‚ 51.44921875% disk.nvme0c0n1.out‚10928.94921875) proc.memory.availableMB‚ 4475.87890625 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.078125 disk.sda2.in‚0 disk.drbd2266.in‚0 network.sent‚ 2.272038e+06 disk.sda2.out‚0$ memory_percent‚72.68140316009521* proc.memory.percent‚18.526768684387207 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 51.44921875# proc.memory.rssMB‚ 3035.42578125 disk.dm-0.out‚10928.94921875) disk./.usagePercent‚18.15301091075438 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 0.919921875 disk.nvme0c0n1.in‚4.078125 disk.sda.in‚0'¾ˆ“ü ‚(:ˆ ÷ˆåԯ뭵 gpu.0.gpu‚81.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚179.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Š©Oý ‚(:Ä ÷ˆåÔާ⸠disk.sda2.out‚0 disk.dm-0.in‚4.078125 disk.sda3.out‚ 52.05078125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 4476.0859375 network.sent‚ 2.278039e+06 disk.rbd1.out‚ 0.919921875 disk.sda1.in‚0 disk.sda.out‚ 52.05078125# proc.memory.rssMB‚ 3035.42578125 disk.rbd0.out‚0% disk.nvme0n1.out‚10968.8916015625 disk.sda2.in‚0 disk.dm-1.out‚ 52.05078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.263175951168165 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.078125 disk.drbd2080.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚18.15079158144005 disk.dm-1.in‚0' disk.nvme0c0n1.out‚10968.8916015625" disk.dm-0.out‚10968.8916015625* proc.memory.percent‚18.526768684387207% disk./.usageGB‚270.45991134643555 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0$ memory_percent‚72.68013954162598 network.recv‚549156 disk.nvme0n1.in‚4.078125\Þg>2þ - _wandbruntime‚1413  _runtime‚1413ó%ˆÇÿ Á % _timestamp‚1.7884294348372583e+09 _runtime‚1413.192302671 _step‚111 & lr-AdamW/pg1‚0.0002044165603871526 & lr-AdamW/pg2‚0.0002044165603871526 trainer/global_step‚1399p­¶:ˀŠ% _timestamp‚1.7884294348372583e+09 _runtime‚1413.192302671 _step‚111 & lr-AdamW/pg1‚0.0002044165603871526 & lr-AdamW/pg2‚0.0002044165603871526 trainer/global_step‚1399o,^å¾2- _wandbruntime‚1413  _runtime‚1413Y~V1‚óO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166336239288320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2985196903927 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192767812510720 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25927332 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192764680726528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189530220055683 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4160044763648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192764680726528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2990250185728 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2987144793088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10960781 ICDeviceStatsMonitor.on_train_batch_start/oversiz­ÅNâe_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2990250185728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55159612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚189777535933440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30121421 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186548126620672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189777562324992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14967062 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2982093435011 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25038191 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186548152048640 _runtime‚1413.193258365 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55136502 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25038185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25927846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166336156138496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162176139509248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189777562324992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14967065 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10960270 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30098317 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25038191 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104pÚ½ý'ƒ÷O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55136502 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25038185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25927846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166336156138496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162176139509248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189777562324992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14967065 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10960270 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30098317 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25038191 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166336239288320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2985196903927 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192767812510720 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25927332 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192764680726528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189530220055683 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4160044763648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192764680726528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2990250185728 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2987144793088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10960781 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2990250185728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55159612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚189777535933440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30121421 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186548126620672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189777562324992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14967062 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2982093435011 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25038191 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186548152048640 _runtime‚1413.193258365 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0p€ôn2„- _wandbruntime‚1414  _runtime‚1414‚$àA'…þM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14977318 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55198012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25055469 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30119407 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55198012 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚189909030021632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2989257106944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987305697235 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2984202228111 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2992362512896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55174876 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10968259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166451513129984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884294357274222e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25055469 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10967758 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25945073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14977315 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162288550373376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚189909030021632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55174876 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30119407 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55198012 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚189909030021632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2989257106944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987305697235 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2984202228111 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2992362512896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55174876 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10968259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166451513129984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejectåÍm@ ions‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884294357274222e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25055469 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10967758 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25945073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14977315 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162288550373376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚189909030021632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55174876 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11342052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193018002109440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25902385 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196352240343040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15479712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196352266734592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199448147026944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26821767 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196352240343040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172103802990080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31179273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57058554 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25902385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172103719840256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31156169 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26821227 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196105523362691 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199445015242752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3095880292352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199448147026944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15479715 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚1449 % _timestamp‚1.7884294821344697e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4307120274432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3095880292352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57081664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11342052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193018002109440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25902385 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196352240343040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15479712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196352266734592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199448147026944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26821767 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196352240343040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172103802990080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31179273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57058554 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25902385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172103719840256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31156169 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26821227 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196105523362691 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199445015242752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3095880292352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199448147026944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15479715 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚1449 % _timestamp‚1.7884294821344697e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4307120274432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3095880292352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57081664 I2–- _wandbruntime‚1461  _runtime‚1461ôÅC’—þM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3094887213568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196483760822784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚1461.386703787 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31200389 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167909065445376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26839682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193147399110656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196483734431232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31177259 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57120064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25919669 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196483734431232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25919675 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193147373682688 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31200389 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26839152 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57096928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31177259 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310052659200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196237029157007 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚1449 % _timestamp‚1.7884294830310173e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3097992619520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3094887213568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11349187 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3089655474319 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11349714 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3097992619520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15489965 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309997657088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199578621644800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57120064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25919669 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167909093579776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088ã",/'˜‚N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167909093579776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3094887213568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196483760822784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚1461.386703787 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31200389 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167909065445376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26839682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193147399110656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196483734431232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31177259 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57120064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25919669 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196483734431232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25919675 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193147373682688 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31200389 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26839152 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57096928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31177259 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310052659200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196237029157007 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚1449 % _timestamp‚1.7884294830310173e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3097992619520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3094887213568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11349187 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3089655474319 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11349714 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3097992619520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15489965 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309997657088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199578621644800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57120064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25919669upèÛ…2™- _wandbruntime‚1461  _runtime‚1461$ì‘TšÎ trainer/global_step‚1449 $ _timestamp‚1.788429483032827e+09 1loss/SRqqt4/assignment_loss‚2.569691399112344 1loss/SRqqt2/detection_loss‚2.6665817201137543 /loss/FBt1/detection_loss‚1.8686402920633554 .loss/FBt2/detection_loss‚3.735890105366707 1loss/SRqqt2/assignment_loss‚2.293530431389809 2loss/SRqqt3/assignment_loss‚1.6340798499186833 1loss/FBt4/assignment_loss‚0.43362083106088295 -loss/FRt2/detection_loss‚2.27824876230696 _step‚118 /loss/FRt4/detection_loss‚3.4854643909554732 /loss/FBt3/detection_loss‚3.0334681164134634 .loss/FBt4/detection_loss‚3.490166307382727 /loss/FRt2/assignment_loss‚5.030935349671737 2loss/SRqqt1/assignment_loss‚1.9297684309422038 0loss/FBt2/assignment_loss‚0.5335064847022286 1loss/FBt3/assignment_loss‚0.49064934698915086 _runtime‚1461.387762356 /loss/FRt1/assignment_loss‚3.249827947656987 %loss/total_loss‚58.65781610370101 0loss/FRt3/assignment_loss‚3.4523668438196182 /loss/FRt4/assignment_loss‚6.005731457158139 1loss/SRqqt1/detection_loss‚1.3853709454415366 0loss/SRqqt4/detection_loss‚3.279990965500474 epoch‚1 0loss/FBt1/assignment_loss‚0.2681054745433454 .loss/FRt1/detection_loss‚2.110688198420961 .loss/FRt3/detection_loss‚1.923587014277776 1loss/SRqqt3/detection_loss‚1.5079054384926955n{ÇX›Ò .loss/FRt3/detection_loss‚1.923587014277776 1loss/SRqqt3/detection_loss‚1.5079054384926955 trainer/global_step‚1449 $ _timestamp‚1.788429483032827e+09 1loss/SRqqt4/assignment_loss‚2.569691399112344 1loss/SRqqt2/detection_loss‚2.6665817201137543 /loss/FBt1/detection_loss‚1.8686402920633554 .loss/FBt2/detection_loss‚3.735890105366707 1loss/SRqqt2/assignment_loss‚2.293530431389809 2loss/SRqqt3/assignment_loss‚1.6340798499186833 1loss/FBt4/assignment_loss‚0.43362083106088295 -loss/FRt2/detection_loss‚2.27824876230696 _step‚118 /loss/FRt4/detection_loss‚3.4854643909554732 /loss/FBt3/detection_loss‚3.0334681164134634 .loss/FBt4/detection_loss‚3.490166307382727 /loss/FRt2/assignment_loss‚5.030935349671737 2loss/SRqqt1/assignment_loss‚1.9297684309422038 0loss/FBt2/assignment_loss‚0.5335064847022286 1loss/FBt3/assignment_loss‚0.49064934698915086 _runtime‚1461.387762356 /loss/FRt1/assignment_loss‚3.249827947656987 %loss/total_loss‚58.65781610370101 0loss/FRt3/assignment_loss‚3.4523668438196182 /loss/FRt4/assignment_loss‚6.005731457158139 1loss/SRqqt1/detection_loss‚1.3853709454415366 0loss/SRqqt4/detection_loss‚3.279990965500474 epoch‚1 0loss/FBt1/assignment_loss‚0.2681054745433454 .loss/FRt1/detection_loss‚2.110688198420961vhMÎ’œ‚(:‡ ³‰åÔÔÿœµ gpu.0.gpu‚78.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚303.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.693 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïOG‚(:¼ ³‰åÔ·šÙ·& disk.nvme0n1.out‚11137.28271484375* disk./.usagePercent‚18.152767191581784 disk.rbd1.out‚ 0.919921875 disk.nvme0c0n1.in‚4.078125( disk.nvme0c0n1.out‚11137.28271484375# disk.dm-0.out‚11137.28271484375 disk.sda3.out‚54.46875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚4.078125 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 cpu‚25.254496383451684 network.sent‚ 2.404294e+06 disk.sda1.out‚0' proc.memory.availableMB‚ 4475.703125$ disk./.usageGB‚270.4893493652344 disk.sda3.in‚0 disk.sda.out‚54.46875 disk.dm-0.in‚4.078125 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚18.526792526245117 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚54.46875$ memory_percent‚72.68247604370117" proc.memory.rssMB‚ 3035.4296875 proc.cpu.threads‚17 network.recv‚568274 disk.drbd2266.in‚0–‡@žž‚(:“ ‰åÔù†œµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.004200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ú²8WŸ‚(:Ì ‰åÔÖÇý¶ disk.dm-0.in‚ 162.1796875 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11192.55224609375 disk.sda1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚11192.55224609375" proc.memory.rssMB‚ 3035.4296875 proc.cpu.threads‚17* disk./.usagePercent‚18.154651151026258 disk.sda1.out‚0 disk.drbd2080.out‚0$ memory_percent‚99.94797706604004* proc.memory.percent‚18.526792526245117 network.recv‚574422 disk.sda2.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚11192.55224609375 disk.sda3.in‚0 network.sent‚ 2.512721e+06 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 54.79296875 disk.dm-1.out‚ 54.79296875 disk.rbd1.out‚ 0.919921875 disk.drbd1716.in‚0 disk.drbd2080.in‚0% proc.memory.availableMB‚ 8.5234375 cpu‚25.252490115301793$ disk./.usageGB‚270.5174217224121! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda.out‚ 54.79296875 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0Fg#¿“ ‚(:ˆ щåÔ¤»µ gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚284.881' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.9762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßV®U¡‚(:Ê щåÔ¨–· disk.rbd0.out‚0( disk.nvme0c0n1.out‚11247.81005859375 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875" proc.memory.rssMB‚ 3035.4296875 disk.drbd1716.out‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.sent‚ 2.519016e+06 disk.sda1.in‚0 disk.sda.out‚ 55.52734375 disk.drbd2266.in‚0 disk.sda3.out‚ 55.52734375$ memory_percent‚99.94802474975586$ proc.memory.availableMB‚8.515625* proc.memory.percent‚18.526792526245117 disk.drbd2080.in‚0 disk.sda2.out‚0# disk.dm-0.out‚11247.81005859375 cpu‚25.25049261469642& disk.nvme0n1.out‚11247.81005859375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 55.52734375 network.recv‚578972 disk.rbd0.in‚0 disk.sda3.in‚0* disk./.usagePercent‚18.154146304168737$ disk./.usageGB‚270.5098991394043 disk.drbd1716.in‚0 disk.drbd2080.out‚0‹ÙÇ2¢- _wandbruntime‚1507  _runtime‚1507`xðȣ $ _timestamp‚1.788429529353656e+09 _runtime‚1507.708697156 _step‚119 ' lr-AdamW/pg1‚0.00020421209324307518 ' lr-AdamW/pg2‚0.00020421209324307518 trainer/global_step‚1499vâ|ÚÌ¤Æ trainer/global_step‚1499 $ _timestamp‚1.788429529353656e+09 _runtime‚1507.708697156 _step‚119 ' lr-AdamW/pg1‚0.00020421209324307518 ' lr-AdamW/pg2‚0.00020421209324307518wpųÝ2¥- _wandbruntime‚1507  _runtime‚1507`Š\Š¦ôO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4453751979008 _runtime‚1507.709900808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206125349758976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27711924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206128481543168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206125349758976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3201510398976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177870949766656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199487826742272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3196103396343 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202926971144192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202680826669699 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32237125 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26766591 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202926971144192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26766585 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202926944752640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3198405006336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32237125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11720103 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15992362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 trainer/global_step‚1499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4453809091584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199487852170240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884295293545132e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26766591 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32214021 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_/νpbatch_start/active_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202683955566583 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27712468 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59003716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15992365 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚173417225922048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3201510398976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚173417197787648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177871035013632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202926944752640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3198405006336 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 trainer/global_step‚1499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4453809091584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199487852170240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884295293545132e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26766591 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32214021 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202683955566583 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27712468 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59003716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15992365 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚173417225922048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3201510398976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚173417197787648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177871035013632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202926944752640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3198405006336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206125349758976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3201510398976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177870949766656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199487826742272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3196103396343 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202926971144192xB\ßO2¨- _wandbruntime‚1508  _runtime‚1508Rt '©þM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206262087958528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206262087958528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59042116 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚203058438840832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59018980 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3203622726144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3200517320192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32258241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203058465232384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 _runtime‚1508.606521569 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3195108720527 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚203058465232384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27730328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16002615 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4456674708480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202815461361107 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚173529608651776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59042116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177986283360256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32235111 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3198212189651 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚1499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11727710 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3203622726144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199617249171456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26783869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11727180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26783875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16002618 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26783875 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32235111 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3200517320192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27729795 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088óæñ¼9"ª‚N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚1499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11727710 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3203622726144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199617249171456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26783869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11727180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26783875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16002618 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26783875 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32235111 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3200517320192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27729795 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206262087958528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206262087958528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59042116 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚203058438840832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59018980 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3203622726144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3200517320192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32258241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203058465232384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 _runtime‚1508.606521569 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3195108720527 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚203058465232384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27730328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16002615 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4456674708480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202815461361107 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚173529608651776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocaÀ·x„Ïtions.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59042116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177986283360256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32235111 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3198212189651 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984y6+F2«- _wandbruntime‚1508  _runtime‚1508ÄßH>X¬Ò 1loss/SRqqt1/detection_loss‚1.7650557852560473 /loss/FBt4/detection_loss‚1.7545324517508838 epoch‚1 -loss/FBt2/detection_loss‚2.60910826921463 % _timestamp‚1.7884295302525442e+09 /loss/FRt2/assignment_loss‚4.724744123571059 0loss/SRqqt4/detection_loss‚2.946531933087569 0loss/FRt4/assignment_loss‚5.6930014115792735 0loss/FBt2/assignment_loss‚0.5696242513751789 /loss/FRt3/detection_loss‚1.8464906197875293 .loss/FRt4/detection_loss‚3.065776065543846 /loss/FBt1/detection_loss‚1.6434454634461715 _runtime‚1508.607541565 2loss/SRqqt1/assignment_loss‚1.8952245121040652 0loss/SRqqt2/assignment_loss‚2.16130821201308 /loss/FRt3/assignment_loss‚2.919243070616651 0loss/FBt4/assignment_loss‚0.5350364438732367 1loss/SRqqt2/detection_loss‚2.8750860341664017 &loss/total_loss‚54.782449086725265 trainer/global_step‚1499 2loss/SRqqt4/assignment_loss‚2.8379527101149926 1loss/SRqqt3/detection_loss‚1.4836298429671628 2loss/SRqqt3/assignment_loss‚1.3929270527521034 /loss/FBt3/detection_loss‚3.8944922387599945 _step‚122 /loss/FRt2/detection_loss‚2.4594441231559303 /loss/FRt1/assignment_loss‚3.286084850629171 0loss/FBt1/assignment_loss‚0.2471880255233496 1loss/FBt3/assignment_loss‚0.10715604887809249 /loss/FRt1/detection_loss‚2.0693655465588425®7Ÿ\­Ö .loss/FRt4/detection_loss‚3.065776065543846 /loss/FBt1/detection_loss‚1.6434454634461715 _runtime‚1508.607541565 2loss/SRqqt1/assignment_loss‚1.8952245121040652 0loss/SRqqt2/assignment_loss‚2.16130821201308 /loss/FRt3/assignment_loss‚2.919243070616651 0loss/FBt4/assignment_loss‚0.5350364438732367 1loss/SRqqt2/detection_loss‚2.8750860341664017 &loss/total_loss‚54.782449086725265 trainer/global_step‚1499 2loss/SRqqt4/assignment_loss‚2.8379527101149926 1loss/SRqqt3/detection_loss‚1.4836298429671628 2loss/SRqqt3/assignment_loss‚1.3929270527521034 /loss/FBt3/detection_loss‚3.8944922387599945 _step‚122 /loss/FRt2/detection_loss‚2.4594441231559303 /loss/FRt1/assignment_loss‚3.286084850629171 0loss/FBt1/assignment_loss‚0.2471880255233496 1loss/FBt3/assignment_loss‚0.10715604887809249 /loss/FRt1/detection_loss‚2.0693655465588425 1loss/SRqqt1/detection_loss‚1.7650557852560473 /loss/FBt4/detection_loss‚1.7545324517508838 epoch‚1 -loss/FBt2/detection_loss‚2.60910826921463 % _timestamp‚1.7884295302525442e+09 /loss/FRt2/assignment_loss‚4.724744123571059 0loss/SRqqt4/detection_loss‚2.946531933087569 0loss/FRt4/assignment_loss‚5.6930014115792735 0loss/FBt2/assignment_loss‚0.5696242513751789 /loss/FRt3/detection_loss‚1.8464906197875293zdnU“®‚(:ˆ à‰åÔÆ›ýµ gpu.0.gpu‚86.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÇ«(X¯‚(:Í à‰åԦī·* disk./.usagePercent‚18.156275006815946 disk.rbd1.in‚0# disk.dm-0.out‚11295.28662109375# memory_percent‚99.9469518661499 cpu‚25.248373313864573 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 55.83203125( disk.nvme0c0n1.out‚11295.28662109375 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda3.out‚ 55.83203125 disk.drbd2080.in‚0 network.recv‚582667 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.drbd1716.in‚0" proc.memory.rssMB‚ 3035.4296875 network.sent‚ 2.524282e+06% disk./.usageGB‚270.54161834716797 disk.sda.out‚ 55.83203125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚18.526792526245117 disk.rbd0.out‚0& disk.nvme0n1.out‚11295.28662109375! disk.nvme0c0n1.in‚ 162.1796875 disk.sda3.in‚0& proc.memory.availableMB‚ 8.69140625 disk.sda1.out‚0 9°‚(:’ ï‰åÔç ´µ gpu.0.gpu‚59.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚301.509' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.30180000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óqvL±‚(:Á ï‰åÔ“Ô¢· disk.drbd2266.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd1716.in‚0* disk./.usagePercent‚18.156943698495382 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 56.5703125 disk.dm-0.in‚ 162.1796875 disk.sda1.out‚0" proc.memory.rssMB‚ 3035.4296875 disk.sda.out‚ 56.5703125 disk.rbd1.out‚ 0.919921875% disk.nvme0c0n1.out‚11352.62890625$ memory_percent‚99.95038509368896 disk.drbd2080.in‚0# disk.nvme0n1.out‚11352.62890625 disk.sda1.in‚0 cpu‚25.246462493892587 disk.drbd2265.in‚0 disk.dm-0.out‚11352.62890625 disk.sda3.in‚0* proc.memory.percent‚18.526792526245117 disk.rbd0.out‚0 disk.drbd1716.out‚0& proc.memory.availableMB‚ 8.12890625 network.sent‚ 2.632712e+06 network.recv‚588772 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 56.5703125$ disk./.usageGB‚270.5515823364258 disk.sda2.in‚0Û¤Ö÷“²‚(:ˆ þ‰åÔ‹Áæ´ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚303.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤_˶Š_runtime‚1555.042183544 _step‚123 & lr-AdamW/pg1‚0.0002040779857969677 & lr-AdamW/pg2‚0.0002040779857969677 trainer/global_step‚1549 % _timestamp‚1.7884295766871538e+09{ª%YT2·- _wandbruntime‚1555  _runtime‚1555Ñèã›ú'¸ôO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16505015 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33294977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179037740994048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212805684275200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4600462232576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307140505600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12099537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚1549 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27630791 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209501649162240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12099000 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3304035112960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚212808816059392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205957702231040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33271873 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183638146114048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27630785 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27630791 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28604552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28604012 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209501675553792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209501675553792¼ñ–£þ'¹øO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚1549 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27630791 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209501649162240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12099000 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3304035112960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚212808816059392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205957702231040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33271873 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183638146114048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27630785 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27630791 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28604552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28604012 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209501675553792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209501675553792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16505015 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33294977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179037740994048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212805684275200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4600462232576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307140505600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12099537|(Ñ*2º- _wandbruntime‚1555  _runtime‚1555ÙÎp“ò »þM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33316093 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28621709 WCDeviceStatsiP<šMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209633143250432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209390764668115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209387635771023 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16515265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60941032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16515268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28622233 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212942422474752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212939290677248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60964168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206087099232256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3306147426816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179150179992576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 % _timestamp‚1.7884295775862677e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183753481567744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212942422474752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60964168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27648075 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33292963 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4603329709568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1555.942146296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27648069 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3306147426816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209390764668115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209387635771023 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16515265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60941032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16515268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28622233 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212942422474752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212939290677248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60964168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206087099232256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3306147426816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179150179992576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 % _timestamp‚1.7884295775862677e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183753481567744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212942422474752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60964168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27648075 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33292963 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4603329709568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1555.942146296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27648069 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3306147426816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28621709 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209633143250432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864}Ó02½- _wandbruntime‚1555  _runtime‚1555+z#ÏX¾Ò 2loss/SRqqt4/assignment_loss‚2.6598606394095854 .loss/FRt4/detection_loss‚3.249118518829346 1loss/FBt1/assignment_loss‚0.19171735728526637 0loss/FBt2/assignment_loss‚0.5952356569353014 /loss/FBt3/assignment_loss‚0.665239441394806 0loss/FBt4/assignment_loss‚0.2419416118298612 /loss/FRt1/detection_loss‚1.9903117669007135 0loss/SRqqt2/detection_loss‚2.551334736808654 1loss/SRqqt4/detection_loss‚3.0781671594489706 /loss/FBt4/detection_loss‚1.4849202846655696 /loss/FRt3/detection_loss‚2.4047811837757336 0loss/SRqqt1/detection_loss‚1.645561703226783 .loss/FBt2/detection_loss‚3.174702413380146 _runtime‚1555.943249609 &loss/total_loss‚57.208616010441304 % _timestamp‚1.7884295775882778e+09 .loss/FRt2/detection_loss‚2.569235993969825 0loss/FRt1/assignment_loss‚2.9180024606840953 0loss/FRt3/assignment_loss‚3.7219869973612765 2loss/SRqqt3/assignment_loss‚1.6232405894861361 epoch‚1 trainer/global_step‚1549 /loss/FRt2/assignment_loss‚4.849753076030362 2loss/SRqqt1/assignment_loss‚1.8796921385960146 1loss/SRqqt3/detection_loss‚1.5397875102029905 /loss/FBt1/detection_loss‚1.7326419433070854 .loss/FBt3/detection_loss‚4.406668663024902 _step‚126 /loss/FRt4/assignment_loss‚5.668579387664795 2loss/SRqqt2/assignment_loss‚2.3661347762230904àá*K\¿Ö &loss/total_loss‚57.208616010441304 % _timestamp‚1.7884295775882778e+09 .loss/FRt2/detection_loss‚2.569235993969825 0loss/FRt1/assignment_loss‚2.9180024606840953 0loss/FRt3/assignment_loss‚3.7219869973612765 2loss/SRqqt3/assignment_loss‚1.6232405894861361 epoch‚1 trainer/global_step‚1549 /loss/FRt2/assignment_loss‚4.849753076030362 2loss/SRqqt1/assignment_loss‚1.8796921385960146 1loss/SRqqt3/detection_loss‚1.5397875102029905 /loss/FBt1/detection_loss‚1.7326419433070854 .loss/FBt3/detection_loss‚4.406668663024902 _step‚126 /loss/FRt4/assignment_loss‚5.668579387664795 2loss/SRqqt2/assignment_loss‚2.3661347762230904 2loss/SRqqt4/assignment_loss‚2.6598606394095854 .loss/FRt4/detection_loss‚3.249118518829346 1loss/FBt1/assignment_loss‚0.19171735728526637 0loss/FBt2/assignment_loss‚0.5952356569353014 /loss/FBt3/assignment_loss‚0.665239441394806 0loss/FBt4/assignment_loss‚0.2419416118298612 /loss/FRt1/detection_loss‚1.9903117669007135 0loss/SRqqt2/detection_loss‚2.551334736808654 1loss/SRqqt4/detection_loss‚3.0781671594489706 /loss/FBt4/detection_loss‚1.4849202846655696 /loss/FRt3/detection_loss‚2.4047811837757336 0loss/SRqqt1/detection_loss‚1.645561703226783 .loss/FBt2/detection_loss‚3.174702413380146 _runtime‚1555.943249609~"è´TžÀ‚(:“ ŠåÔš¯¼µ gpu.0.gpu‚92.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚158.485' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.697000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚$â]TÁ‚(:É ŠåÔä¹ñ· disk.rbd1.in‚0 disk.sda.out‚ 57.64453125 proc.cpu.threads‚17 network.sent‚ 2.644996e+06 disk.sda2.out‚0" disk.dm-0.out‚11460.2900390625 disk.sda3.out‚ 57.64453125 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚99.94940757751465* proc.memory.percent‚18.526816368103027 disk.drbd2265.in‚0 disk.dm-0.in‚ 162.1796875 disk.sda1.out‚0 network.recv‚597780$ disk./.usageGB‚270.5950813293457# proc.memory.rssMB‚ 3035.43359375 disk.drbd2265.out‚0% disk.nvme0n1.out‚11460.2900390625 disk.rbd1.out‚ 0.919921875 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0% proc.memory.availableMB‚ 8.2890625* disk./.usagePercent‚18.159862952408325 disk.rbd0.out‚0 disk.drbd2266.in‚0! disk.nvme0c0n1.in‚ 162.1796875' disk.nvme0c0n1.out‚11460.2900390625 disk.sda1.in‚0 disk.sda.in‚0 cpu‚25.24256804176731 disk.dm-1.in‚0 disk.dm-1.out‚ 57.64453125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.in‚0ô}ޞ‚(:“ œŠåÔâÿƒµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.839200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†²¥YXÂ(:Í œŠåÔšÿ¹·& proc.memory.availableMB‚ 8.26171875 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚11507.81982421875 disk.rbd0.out‚0 disk.dm-1.out‚ 58.06640625 disk.drbd2266.in‚0# disk.dm-0.out‚11507.81982421875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 cpu‚25.240777609817698 disk.drbd2265.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚11507.81982421875! disk.nvme0c0n1.in‚ 162.1796875 disk.sda1.in‚0$ disk./.usageGB‚270.6012954711914 disk.sda.out‚ 58.06640625 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 162.1796875 disk.sda1.out‚0* disk./.usagePercent‚18.160279988681598 disk.sda.in‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 3035.43359375* proc.memory.percent‚18.526816368103027 network.sent‚ 2.75286e+06 network.recv‚603298 disk.rbd0.in‚0 disk.rbd1.out‚ 0.919921875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 58.06640625 disk.drbd2080.out‚0$ memory_percent‚99.94957447052002í9.?Ä‚(:’ «ŠåÔ»®µ gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m@X¥TÅ‚(:É «ŠåÔÓÌ ·$ disk./.usageGB‚270.6110038757324 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 58.7265625 disk.drbd2080.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚11567.78466796875* proc.memory.percent‚18.526816368103027 network.recv‚607979 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 8.0078125* disk./.usagePercent‚18.160931527856242 disk.sda.out‚ 58.7265625 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.drbd1716.out‚0& disk.nvme0n1.out‚11567.78466796875 disk.dm-1.out‚ 58.7265625 disk.rbd1.out‚ 0.919921875 proc.cpu.threads‚17 network.sent‚ 2.758799e+06 disk.sda.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.in‚0$ memory_percent‚99.95112419128418# proc.memory.rssMB‚ 3035.43359375 disk.sda1.out‚0 disk.drbd1716.in‚0# disk.dm-0.out‚11567.78466796875 cpu‚25.23901332677491@–2Æ- _wandbruntime‚1602  _runtime‚1602§íÏnÈÇ ' lr-AdamW/pg2‚0.00020392266488964276 trainer/global_step‚1599 $ _timestamp‚1.788429624334499e+09 _runtime‚1602.689539653 _step‚127 ' lr-AdamW/pg1‚0.00020392266488964276£æÌÌÈÆ _step‚127 ' lr-AdamW/pg1‚0.00020392266488964276 ' lr-AdamW/pg2‚0.00020392266488964276 trainer/global_step‚1599 $ _timestamp‚1.788429624334499e+09 _runtime‚1602.689539653äì©2É- _wandbruntime‚1602  _runtime‚1602s¶6¿ÊôO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219489150575616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12478642 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189405409349120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚216076353571840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62847820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34352829 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184658312334848 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219486018791424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚216076379963392 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 _step‚128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚1602.690546413 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34352829 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12478116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34329725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3412770612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215831433283715 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17017665 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3409665219584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215834562180599 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28494985 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17017662 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 trainer/global_step‚1599 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚216076353571840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34329725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4747125148672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28494991 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212427526863872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3412770612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29496307 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212427552291840 % _timestamp‚1.7884296243352416e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62824710 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29495778 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 cQDeviceStatsóÎ; Monitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4747180164096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184658284200448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3403906419843 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3409665219584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189405492498944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3407009888759 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚216076379963392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180ÐÔÇõÿ'ËùO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3409665219584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189405492498944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3407009888759 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚216076379963392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219489150575616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12478642 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189405409349120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚216076353571840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62847820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34352829 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184658312334848 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219486018791424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚216076379963392 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 _step‚128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚1602.690546413 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34352829 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12478116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34329725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3412770612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215831433283715 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17017665 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3409665219584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215834562180599 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28494985 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17017662 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 trainer/global_step‚1599 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚216076353571840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34329725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4747125148672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28494991 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212427526863872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3412770612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29496307 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212427552291840 % _timestamp‚1.7884296243352416e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62824710 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29495778 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4747180164096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184658284200448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3403906419843 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0€¨~Ðd2Ì- _wandbruntime‚1603  _runtime‚1603îÀþž'ÍþM WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34373945 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215962939078031 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3406015212943 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34350815 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212556949293056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3409118682067 _runtime‚1603.591608295 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28512269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚216207847660032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4750058307072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1599 _step‚129 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28512269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34373945 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3414882939392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184770695064576 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3411777533440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62863084 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62863084 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚219622756990976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4750113309184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215966067975123 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚219622756990976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17027918 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189520836508160 % _timestamp‚1.7884296252360837e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34350815 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62886220 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3411777533440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29513615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184770723198976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62886220 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180jIã’ '΃N REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34373945 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215962939078031 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3406015212943 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34350815 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212556949293056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3409118682067 _runtime‚1603.591608295 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28512269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚216207847660032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4750058307072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1599 _step‚129 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28512269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34373945 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3414882939392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184770695064576 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3411777533440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62863084 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62863084 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚219622756990976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4750113309184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215966067975123 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚219622756990976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17027918 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189520836508160 % _timestamp‚1.7884296252360837e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34350815 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62886220 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3411777533440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29513615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184770723198976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62886220 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552Ü I.2Ï- _wandbruntime‚1603  _runtime‚1603eX 5$ÐÓ .loss/FRt2/detection_loss‚…DM52.417797384943281 epoch‚1 0loss/FRt3/assignment_loss‚3.8780074862011693 /loss/FRt3/detection_loss‚2.1264956583056533 .loss/FBt3/detection_loss‚3.708964467048645 $ _timestamp‚1.788429625237677e+09 _runtime‚1603.592642327 0loss/FBt3/assignment_loss‚0.7936776727437973 1loss/SRqqt4/detection_loss‚2.5505262226970107 trainer/global_step‚1599 /loss/FBt4/detection_loss‚1.8893317758637684 1loss/SRqqt2/assignment_loss‚1.845849061012268 2loss/SRqqt3/assignment_loss‚1.4254001639783382 1loss/FBt1/assignment_loss‚0.30396705537806873 1loss/FBt4/assignment_loss‚0.45622510564428576 /loss/FRt4/detection_loss‚3.2423097954856024 /loss/FBt2/detection_loss‚3.2336763739585876 0loss/FRt1/assignment_loss‚3.1303796754641966 1loss/SRqqt4/assignment_loss‚2.717786584739332 /loss/SRqqt3/detection_loss‚1.35437623754646 /loss/FBt1/detection_loss‚1.8144597740636932 _step‚130 /loss/FRt1/detection_loss‚2.3260288042100994 1loss/SRqqt1/detection_loss‚1.5855321621424274 %loss/total_loss‚56.45489400421633 0loss/FRt2/assignment_loss‚5.1630941663469585 0loss/FBt2/assignment_loss‚0.6043164525180913 /loss/FRt4/assignment_loss‚5.936723011511344 1loss/SRqqt2/detection_loss‚2.3470136066277822 2loss/SRqqt1/assignment_loss‚1.6029553057854635©am^ÑØ 2loss/SRqqt1/assignment_loss‚1.6029553057854635 .loss/FRt2/detection_loss‚2.417797384943281 epoch‚1 0loss/FRt3/assignment_loss‚3.8780074862011693 /loss/FRt3/detection_loss‚2.1264956583056533 .loss/FBt3/detection_loss‚3.708964467048645 $ _timestamp‚1.788429625237677e+09 _runtime‚1603.592642327 0loss/FBt3/assignment_loss‚0.7936776727437973 1loss/SRqqt4/detection_loss‚2.5505262226970107 trainer/global_step‚1599 /loss/FBt4/detection_loss‚1.8893317758637684 1loss/SRqqt2/assignment_loss‚1.845849061012268 2loss/SRqqt3/assignment_loss‚1.4254001639783382 1loss/FBt1/assignment_loss‚0.30396705537806873 1loss/FBt4/assignment_loss‚0.45622510564428576 /loss/FRt4/detection_loss‚3.2423097954856024 /loss/FBt2/detection_loss‚3.2336763739585876 0loss/FRt1/assignment_loss‚3.1303796754641966 1loss/SRqqt4/assignment_loss‚2.717786584739332 /loss/SRqqt3/detection_loss‚1.35437623754646 /loss/FBt1/detection_loss‚1.8144597740636932 _step‚130 /loss/FRt1/detection_loss‚2.3260288042100994 1loss/SRqqt1/detection_loss‚1.5855321621424274 %loss/total_loss‚56.45489400421633 0loss/FRt2/assignment_loss‚5.1630941663469585 0loss/FBt2/assignment_loss‚0.6043164525180913 /loss/FRt4/assignment_loss‚5.936723011511344 1loss/SRqqt2/detection_loss‚2.3470136066277822‚P•1º“Ò‚(:ˆ ºŠåÔ̵ۚ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚205.753' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.1506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ApVÓ‚(:Ë ºŠåÔãçÅ· network.recv‚612260 disk.dm-0.in‚ 162.1796875 disk.sda.out‚ 59.09765625 disk.drbd2266.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.out‚ 59.09765625 proc.cpu.threads‚17% disk./.usageGB‚270.61842346191406 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0$ memory_percent‚99.95005130767822) disk./.usagePercent‚18.16142946251034' disk.nvme0c0n1.out‚11618.3154296875 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 3035.43359375 network.sent‚ 2.764823e+06 disk.sda3.out‚ 59.09765625 disk.drbd1716.out‚0& proc.memory.availableMB‚ 8.18359375 disk.rbd1.out‚ 0.919921875" disk.dm-0.out‚11618.3154296875 disk.sda2.in‚0* proc.memory.percent‚18.526816368103027 cpu‚25.237111995087748% disk.nvme0n1.out‚11618.3154296875 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0Ý¿¿–‘Ô‚(:† ÉŠåÔ©Ù¯µ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚125.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–éÓDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35387577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226166353307648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218897376924672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222651084372992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222651057981440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4894103652864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226169485091840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222651084372992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35387577 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226169485091840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3509359666051 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64769872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195172931059712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3518400718848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12859286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222406736590723 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 % _timestamp‚1.7884296719339826e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4894158668288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 trainer/global_step‚1649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64746762 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29359191 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35410681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17530312 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30390122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190278855541248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190278827406848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456å|è†ÿ'ßùO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226166353307648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218897376924672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222651084372992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222651057981440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4894103652864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226169485091840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222651084372992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35387577 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226169485091840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3509359666051 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64769872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195172931059712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3518400718848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12859286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222406736590723 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 % _timestamp‚1.7884296719339826e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4894158668288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 trainer/global_step‚1649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64746762 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29359191 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35410681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17530312 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30390122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190278855541248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190278827406848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29359191 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218897402352640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29359185 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3515295326208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222409865487607 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _runtime‚1650.289244477 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35387577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180„B^«2à- _wandbruntime‚1651  _runtime‚1651Ž‚ý3 áþM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195288273354752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64808272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚1649 _runtime‚1651.188260719 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64808272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12867312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _step‚133 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222782552069632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29376475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12866816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4897035083776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3514571928275 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3517407640064 REDeviceStatsMonitor.on_train_batch_end/inactiveÕCÎÑ_split.large_pool.freed‚17540565 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226303091507200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222782578461184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30407381 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17540568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222538242385039 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884296728328252e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222541371282131 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3511468459151 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3517407640064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30407880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29376475 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3520513046016 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29376469 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35431797 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222782552069632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64785136 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3520513046016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190391266405376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64785136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52904960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29376469 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4897087988736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35431797!ײ 'âƒN C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64785136 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3520513046016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190391266405376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64785136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52904960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29376469 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4897087988736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35431797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195288273354752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64808272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚1649 _runtime‚1651.188260719 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64808272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12867312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _step‚133 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222782552069632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29376475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12866816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4897035083776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3514571928275 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3517407640064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17540565 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226303091507200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222782578461184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30407381 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17540568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222538242385039 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884296728328252e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222541371282131 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3511468459151 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3517407640064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30407880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29376475 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3520513046016 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29376469 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35431797 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222782552069632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0…6Çá2ã- _wandbruntime‚1651  _runtime‚1651æñ“–XäÒ /loss/FRt2/assignment_loss‚5.234328562021256 0loss/FBt2/assignment_loss‚0.3843642978126809 1loss/SRqqt1/detection_loss‚1.5440732481933774 .loss/FRt1/assignment_loss‚3.06861754616753 2loss/SRqqt3/assignment_loss‚1.4173540965198201 1loss/FBt4/assignment_loss‚0.37042895212461696 /loss/FBt4/detection_loss‚1.7000599232033091 .loss/FRt2/detection_loss‚2.556359389424324 /loss/FBt2/detection_loss‚3.3567617386579514 _step‚134 /loss/FRt1/detection_loss‚1.7948695851154015 2loss/SRqqt1/assignment_loss‚1.8806575409003667 epoch‚1 0loss/FRt3/assignment_loss‚3.2216547983033315 %loss/total_loss‚53.38092949486124 # _timestamp‚1.78842967283459e+09 0loss/FRt4/assignment_loss‚5.5327918815612795 1loss/FBt1/assignment_loss‚0.14602555331811645 0loss/SRqqt4/detection_loss‚2.607027691602707 /loss/FBt3/detection_loss‚3.0341852055146146 1loss/FBt3/assignment_loss‚0.25851240796551933 /loss/FRt3/detection_loss‚2.1154872085603458 1loss/SRqqt2/detection_loss‚3.1306546874668286 _runtime‚1651.189573296 2loss/SRqqt2/assignment_loss‚2.1699861333422037 1loss/SRqqt4/assignment_loss‚2.296850377321243 .loss/FRt4/detection_loss‚3.120371685028076 1loss/SRqqt3/detection_loss‚1.1571791678112238 .loss/FBt1/detection_loss‚1.282327816925115 trainer/global_step‚1649ë^éÙ]å× .loss/FRt2/detection_loss‚2.556359389424324 /loss/FBt2/detection_loss‚3.3567617386579514 _step‚134 /loss/FRt1/detection_loss‚1.7948695851154015 2loss/SRqqt1/assignment_loss‚1.8806575409003667 epoch‚1 0loss/FRt3/assignment_loss‚3.2216547983033315 %loss/total_loss‚53.38092949486124 # _timestamp‚1.78842967283459e+09 0loss/FRt4/assignment_loss‚5.5327918815612795 1loss/FBt1/assignment_loss‚0.14602555331811645 0loss/SRqqt4/detection_loss‚2.607027691602707 /loss/FBt3/detection_loss‚3.0341852055146146 1loss/FBt3/assignment_loss‚0.25851240796551933 /loss/FRt3/detection_loss‚2.1154872085603458 1loss/SRqqt2/detection_loss‚3.1306546874668286 _runtime‚1651.189573296 2loss/SRqqt2/assignment_loss‚2.1699861333422037 1loss/SRqqt4/assignment_loss‚2.296850377321243 .loss/FRt4/detection_loss‚3.120371685028076 1loss/SRqqt3/detection_loss‚1.1571791678112238 .loss/FBt1/detection_loss‚1.282327816925115 trainer/global_step‚1649 /loss/FRt2/assignment_loss‚5.234328562021256 0loss/FBt2/assignment_loss‚0.3843642978126809 1loss/SRqqt1/detection_loss‚1.5440732481933774 .loss/FRt1/assignment_loss‚3.06861754616753 2loss/SRqqt3/assignment_loss‚1.4173540965198201 1loss/FBt4/assignment_loss‚0.37042895212461696 /loss/FBt4/detection_loss‚1.7000599232033091†0‘Úæ‚(:’ öŠåÔŠ¹Êµ gpu.0.gpu‚91.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.324' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.66480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JaÈJç‚(:¿ öŠåÔø°‚¸ disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0" disk.nvme0n1.out‚ 11829.7890625 disk.drbd2266.in‚0 disk.dm-0.out‚ 11829.7890625 disk.sda.out‚ 61.1953125 disk.rbd0.out‚0 disk.rbd1.out‚ 0.919921875 disk.drbd2080.in‚0& proc.memory.availableMB‚ 8.85546875 disk.dm-1.out‚ 61.1953125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 61.1953125 disk.sda1.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 11829.7890625 disk.dm-0.in‚ 162.1796875 disk.drbd1716.out‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 3035.43359375* proc.memory.percent‚18.526816368103027) disk./.usagePercent‚18.16316468157742 disk.rbd1.in‚0$ memory_percent‚99.94595050811768 cpu‚25.230160848263218! disk.nvme0c0n1.in‚ 162.1796875 network.recv‚630417 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 network.sent‚ 2.889939e+06% disk./.usageGB‚270.64427947998047SÏQ>è‚(:’ …‹åÔŠ¨‡µ gpu.0.gpu‚65.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.063' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.41259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”žTJNé‚(:à …‹åÔÒ©Ó· disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875( disk.nvme0c0n1.out‚11873.98193359375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.dm-1.out‚61.78125 disk.rbd1.in‚0# disk.dm-0.out‚11873.98193359375 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.998759e+06) disk./.usagePercent‚18.16512160316915 disk.drbd2265.in‚0 network.recv‚637180& disk.nvme0n1.out‚11873.98193359375 disk.drbd2266.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 3035.43359375 cpu‚25.228588317802775 disk.dm-0.in‚ 162.1796875 disk.drbd1716.out‚0* proc.memory.percent‚18.526816368103027 disk.sda1.in‚0 disk.sda.out‚61.78125$ memory_percent‚99.94750022888184$ disk./.usageGB‚270.6734390258789 disk.drbd2266.out‚0 disk.sda3.out‚61.78125 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 162.1796875% proc.memory.availableMB‚ 8.6015625 disk.nvme0n1.in‚ 162.1796875Ý´9žê‚(:“ ”‹åÔÁÁ”µ gpu.0.gpu‚88.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚286.734' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.346799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aµ¬rOë‚(:Ä ”‹åÔ°¨¾¶ disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.919921875 disk.drbd2265.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.166089823663647 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0# proc.memory.availableMB‚8.84375# proc.memory.rssMB‚ 3035.43359375& disk.nvme0n1.out‚11919.75146484375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚11919.75146484375 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.sda3.out‚ 62.171875 disk.sda.out‚ 62.171875# memory_percent‚99.9460220336914$ disk./.usageGB‚270.6878662109375# disk.dm-0.out‚11919.75146484375 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 cpu‚25.226921586782492 network.sent‚ 3.004007e+06 network.recv‚640658 disk.dm-1.out‚ 62.171875* proc.memory.percent‚18.526816368103027 disk.drbd2266.in‚0 disk.sda2.out‚0òÞ´¬2ì- _wandbruntime‚1697  _runtime‚1697˜enŠÇíÁ & lr-AdamW/pg1‚0.0002035485165800256 & lr-AdamW/pg2‚0.0002035485165800256 trainer/global_step‚1699 % _timestamp‚1.7884297190940375e+09 _runtime‚1697.449078383 _step‚135ÿ·ÌîÆ _step‚135 & lr-AdamW/pg1‚0.0002035485165800256 & lr-AdamW/pg2‚0.0002035485165800256 trainer/global_step‚1699 % _timestamp‚1.7884297190940375e+09 _runtime‚1697.449078383‡†©-æ2ï- _wandbruntime‚1697  _runtime‚1697Q=ywðôO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36445429 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229225762391040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66691924 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30223385 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228982039897731 % _timestamp‚1.7884297190947802e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232846687823872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229225788782592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18042962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36468533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3614812912259 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66668814 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229225788782592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36445429 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13239996 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225367226985472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200940440927232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195899398747648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229225762391040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31283497 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200940526174208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3617916381175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3620925432832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3624030825472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232846687823872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228985168794615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36468533 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3620925432832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5041070313984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13240532 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1699 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30223385 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195899370613248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5041127426560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1697.450008109 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30223391 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31282958 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232846687823872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229225788782592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18042962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36468533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3614812912259 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66668814 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229225788782592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36445429 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13239996 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225367226985472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200940440927232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195899398747648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229225762391040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31283497 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200940526174208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3617916381175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3620925432832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3624030825472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232846687823872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228985168794615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36468533 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3620925432832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5041070313984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13240532 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1699 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30223385 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241ˆÉìþì2ò- _wandbruntime‚1698  _runtime‚1698‡Ð­;'óþM RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30240675 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31300793 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3623037746688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66730324 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66730324 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66707188 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31301314 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚225496649414656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229357256479232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225496623986688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3620025174483 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3626143152640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18053215 % _timestamp‚1.7884297199953735e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _step‚137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229113545692047 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36489649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232983426023424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232983426023424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3623037746688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3616921705359 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229116674589139 trainer/global_step‚1699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229357282870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13248096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196011781477376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _runtime‚1698.350941961 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201055878555648 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196011809611776 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66707188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Z3Ú Ç&ôƒN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229357256479232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30240669 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5044068943872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66730324 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66730324 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66707188 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31301314 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚225496649414656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229357256479232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225496623986688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3620025174483 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3626143152640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18053215 % _timestamp‚1.7884297199953735e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _step‚137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229113545692047 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36489649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232983426023424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232983426023424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3623037746688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3616921705359 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229116674589139 trainer/global_step‚1699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229357282870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13248096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196011781477376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _runtime‚1698.350941961 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201055878555648 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196011809611776 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66707188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30240675 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31300793 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3623037746688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 B•‘ø‚(:† £‹åÔõ¨½µ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚125.301' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åñ&£Yù‚(:Î £‹åÔÛõÿ· network.recv‚644916* disk./.usagePercent‚18.167090557114996 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 62.76171875 disk.sda.out‚ 62.76171875 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 162.1796875 disk.rbd0.in‚0$ memory_percent‚99.94642734527588 proc.cpu.threads‚17 network.sent‚ 3.010047e+06 disk.dm-1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚11980.76416015625 disk.sda2.in‚0 disk.drbd1716.out‚0$ disk./.usageGB‚270.7027778625488 disk.dm-1.out‚ 62.76171875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0& proc.memory.availableMB‚ 8.77734375 disk.rbd1.out‚ 0.919921875 disk.drbd2080.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 3035.43359375* proc.memory.percent‚18.526816368103027! disk.nvme0c0n1.in‚ 162.1796875 disk.sda1.out‚0& disk.nvme0n1.out‚11980.76416015625# disk.dm-0.out‚11980.76416015625 disk.rbd0.out‚0 cpu‚25.225491102544524 disk.drbd2080.in‚0 disk.sda.in‚0‘ ÖÌ‘ú‚(:† ²‹åÔýø’µ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚146.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉF_;Qû‚(:Æ ²‹åÔ¨Þ¸ disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0% disk./.usageGB‚270.67397689819336 disk.drbd2265.in‚0 network.recv‚651012 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875! disk.nvme0c0n1.in‚ 162.1796875 disk.sda.out‚ 63.203125 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0) disk./.usagePercent‚18.16515770023148( disk.nvme0c0n1.out‚12028.88916015625# memory_percent‚99.9464750289917 cpu‚25.22384439533335 network.sent‚ 3.118342e+06 disk.sda1.in‚0 disk.sda.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 63.203125& proc.memory.availableMB‚ 8.76953125 disk.sda3.in‚0 disk.sda3.out‚ 63.203125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.1796875& disk.nvme0n1.out‚12028.88916015625# proc.memory.rssMB‚ 3035.43359375 disk.dm-0.in‚ 162.1796875 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.526816368103027# disk.dm-0.out‚12028.88916015625C”' ü‚(:’ Á‹åÔ”³û´ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.552' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.91040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÍ=Qý‚(:Æ Á‹åÔ¡¼õ¶* disk./.usagePercent‚18.167766673012924 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚8.515625 disk.rbd1.out‚ 0.919921875' disk.nvme0c0n1.out‚12096.5751953125 disk.sda3.out‚ 63.8515625 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.1796875* proc.memory.percent‚18.526816368103027 cpu‚25.22239979009266" disk.dm-0.out‚12096.5751953125 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 63.8515625 disk.sda2.in‚0$ memory_percent‚99.94802474975586 proc.cpu.threads‚17 network.recv‚655881 disk.sda.out‚ 63.8515625 disk.rbd0.in‚0% disk.nvme0n1.out‚12096.5751953125 network.sent‚ 3.124843e+06% disk./.usageGB‚270.71285247802734! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.sda1.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 3035.43359375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0q¸ÿÌ2þ- _wandbruntime‚1744  _runtime‚1744kH—AÉÿà _step‚139 ' lr-AdamW/pg1‚0.00020332976714968035 ' lr-AdamW/pg2‚0.00020332976714968035 trainer/global_step‚1749 % _timestamp‚1.7884297666178336e+09 _runtime‚1744.972877397<ëúžÎ€È _runtime‚1744.972877397 _step‚139 ' lr-AdamW/pg1‚0.00020332976714968035 ' lr-AdamW/pg2‚0.00020332976714968035 trainer/global_step‚1749 % _timestamp‚1.7884297666178336e+09‹¹á72- _wandbruntime‚1744  _runtime‚1744·.•"ú'‚ôO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3729660932096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231837077046272 _step‚140 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚239530154124288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5187659394048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235560472101623 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235800493192192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13620311 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235800466800640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32176438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201519941954048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚239530154124288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31087591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3729660932096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206707656363520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5187714409472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239527022340096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3726555539456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231837102474240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31087585 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68590866 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚1744.973847648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32175923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18555612 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37503281 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37503281 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37526385 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13620823 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3726555539456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37526385 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235800466800640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235557343204739 % _timestamp‚1.7884297666185753e+09ëÂøhÿ'ƒùO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3729660932096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206707656363520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5187714409472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239527022340096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3726555539456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231837102474240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31087585 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68590866 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚1744.973847648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32175923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18555612 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37503281 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37503281 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37526385 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13620823 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3726555539456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37526385 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235800466800640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235557343204739 % _timestamp‚1.7884297666185753e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3729660932096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231837077046272 _step‚140 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚239530154124288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5187659394048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235560472101623 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235800493192192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13620311 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235800466800640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32176438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201519941954048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚239530154124288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31087591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968ŒõsÕú2„- _wandbruntime‚1745  _runtime‚1745âµ?e……ýM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239660628742144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231966474047488 trainer/global_step‚1749 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201632324683776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚141 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32193754 UHDeviceStatsMonitor.oné*~_train_batch_end/requested_bytes.large_pool.current‚25427968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68652376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5190661819392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3725478420691 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37547501 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206822931501056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3731773259264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3728667853312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239663760539648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37524371 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3722374951567 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37524371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31104875 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201632352818176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13627889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32194254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235931960888832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68629240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235691977896147«j'†‚N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206823014637568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231966474047488 trainer/global_step‚1749 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201632324683776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚141 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32193754 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68652376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5190661819392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3725478420691 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37547501 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206822931501056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3731773259264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3728667853312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239663760539648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37524371 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3722374951567 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37524371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31104875 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201632352818176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13627889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32194254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235931960888832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68629240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235691977896147 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239660628742144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5334395557376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19068262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14002078 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246210488640512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31951785 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242375197601792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246207356856320 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31951791 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212474937830400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38584237 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70512918 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14001529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242132646511747 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31951791 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212474852583424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884298138660645e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246210488640512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238306927107072 _runtime‚1792.221333952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3825719404675 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38561133 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246207356856320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242375171210240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3835291038720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242375197601792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStat͸kæŒsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1799 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3832185646080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚242135775408631 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884f†»ÿ'•ùO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38561133 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207140485160448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238306952535040 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19068265 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70536028 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207140457026048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3828822873591 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242375171210240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38584237 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3835291038720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33070343 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5334452669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3832185646080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246210488640512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238306927107072 _runtime‚1792.221333952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3825719404675 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38561133 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246207356856320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242375171210240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3835291038720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242375197601792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1799 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3832185646080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚242135775408631 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5334395557376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19068262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14002078 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246210488640512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31951785 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242375197601792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246207356856320 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31951791 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212474937830400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38584237 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70512918 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14001529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242132646511747 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31951791 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212474852583424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884298138660645e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242506665298432 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38582223 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212590205565952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242506665298432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3834297959936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242264152306063 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1793.128623656 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70551292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207252867890176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3827828197775 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70574428 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242267281203155 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246344095055872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33087383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238436324108288 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70574428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5337394775040 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238436349536256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31969069 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242506691689984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242506691689984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3837403365888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19078515 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38582223 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14008868 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242506691689984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242506691689984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3837403365888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19078515 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38582223 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14008868 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242506665298432 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38582223 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212590205565952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242506665298432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3834297959936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242264152306063 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1793.128623656 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70551292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207252867890176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3827828197775 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70574428 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242267281203155 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246344095055872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33087383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238436324108288 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70574428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5337394775040 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238436349536256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31969069 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33961385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5480748872704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39642089 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252887691372544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39642089 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19580915 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚248711078715639 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3931172650883 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252890823156736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32815991 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39618985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252887691372544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3937815752704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5480693857280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884298609953628e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3940921145344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32815985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218241694089728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32815985 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212761028366848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72434970 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3940921145344 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244776777167872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14381006 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14380473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248949875619840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚244776802595840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248949902011392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19580912 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32815991 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚248707949818755 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚1839.350750607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3937815752704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚148 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890823156736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72458080 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218241777239552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚248949902011392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33961921 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33961385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5480748872704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39642089 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252887691372544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39642089 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19580915 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚248711078715639 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3931172650883 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252890823156736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32815991 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39618985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252887691372544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3937815752704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5480693857280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884298609953628e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3940921145344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32815985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218241694089728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32815985 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212761028366848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72434970 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3940921145344 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244776777167872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14381006 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241”2mém2¨- _wandbruntime‚1840  _runtime‚1840$JÉÖΩþM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212873439230976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32833275 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253021297774592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32833275 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3943033472512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253024429572096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39640075 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248839455613071 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244906199597056 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32833269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3939928066560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚1840.239228672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33979835 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253024429572096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3943033472512 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.õ#´’6large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72473344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72473344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72496480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 trainer/global_step‚1849 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14388139 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218357037836288 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19591165 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39640075 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39663205 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5483681741824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33979304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212873411096576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3939928066560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5483626739712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3933281443983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249081396099584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218357120972800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248842584510163 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72496480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249081396099584 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249081369708032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884298618828871e+09–߯? 'ªƒN cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212873439230976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32833275 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253021297774592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32833275 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3943033472512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253024429572096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39640075 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248839455613071 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244906199597056 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32833269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3939928066560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚1840.239228672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33979835 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253024429572096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3943033472512 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72473344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72473344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72496480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 trainer/global_step‚1849 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14388139 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218357037836288 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19591165 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39640075 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39663205 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5483681741824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33979304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212873411096576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3939928066560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5483626739712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3933281443983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249081396099584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218357120972800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248842584510163 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72496480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249081396099584 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249081369708032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884298618828871e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0•Fp'2«- _wandbruntime‚1840  _runtime‚1840!p¶qX¬Ò 1loss/SRqqt2/assignment_loss‚2.183137355229029 2loss/SRqqt3/assignment_loss‚1.6900259287918316 1loss/SRqqt1/detection_loss‚1.4494065283132451 1loss/SRqqt2/detection_loss‚2.8303102942613454 1loss/SRqqt3/detection_loss‚1.2948522160158438 epoch‚1 .loss/FRt1/detection_loss‚2.408050152049823 %loss/total_loss‚50.86925609532144 1loss/SRqqt4/detection_loss‚2.7878395948145123 0loss/FRt3/assignment_loss‚2.2642964467089226 2loss/SRqqt1/assignment_loss‚1.6725986395031214 0loss/FBt1/assignment_loss‚0.2805862160317335 0loss/FBt3/assignment_loss‚0.2364649710555824 /loss/FRt3/detection_loss‚2.2641590262981173 trainer/global_step‚1849 /loss/FBt1/detection_loss‚1.3504915137642197 /loss/FRt4/detection_loss‚3.2681486237616766 % _timestamp‚1.7884298618855326e+09 _runtime‚ 1840.24055763 .loss/FBt2/detection_loss‚2.845304055346383 /loss/FBt3/detection_loss‚2.7595404336849847 1loss/FBt4/assignment_loss‚0.32387530526847674 /loss/FRt2/detection_loss‚2.4887867149981586 /loss/FRt2/assignment_loss‚4.759185498411005 /loss/FRt4/assignment_loss‚5.201681806927636 2loss/SRqqt4/assignment_loss‚2.1505711387704918 _step‚150 /loss/FRt1/assignment_loss‚1.987051327120174 0loss/FBt2/assignment_loss‚0.5403479181964409 .loss/FBt4/detection_loss‚1.832544389998688.ÚA&]­× 0loss/FBt3/assignment_loss‚0.2364649710555824 /loss/FRt3/detection_loss‚2.2641590262981173 trainer/global_step‚1849 /loss/FBt1/detection_loss‚1.3504915137642197 /loss/FRt4/detection_loss‚3.2681486237616766 % _timestamp‚1.7884298618855326e+09 _runtime‚ 1840.24055763 .loss/FBt2/detection_loss‚2.845304055346383 /loss/FBt3/detection_loss‚2.7595404336849847 1loss/FBt4/assignment_loss‚0.32387530526847674 /loss/FRt2/detection_loss‚2.4887867149981586 /loss/FRt2/assignment_loss‚4.759185498411005 /loss/FRt4/assignment_loss‚5.201681806927636 2loss/SRqqt4/assignment_loss‚2.1505711387704918 _step‚150 /loss/FRt1/assignment_loss‚1.987051327120174 0loss/FBt2/assignment_loss‚0.5403479181964409 .loss/FBt4/detection_loss‚1.832544389998688 1loss/SRqqt2/assignment_loss‚2.183137355229029 2loss/SRqqt3/assignment_loss‚1.6900259287918316 1loss/SRqqt1/detection_loss‚1.4494065283132451 1loss/SRqqt2/detection_loss‚2.8303102942613454 1loss/SRqqt3/detection_loss‚1.2948522160158438 epoch‚1 .loss/FRt1/detection_loss‚2.408050152049823 %loss/total_loss‚50.86925609532144 1loss/SRqqt4/detection_loss‚2.7878395948145123 0loss/FRt3/assignment_loss‚2.2642964467089226 2loss/SRqqt1/assignment_loss‚1.6725986395031214 0loss/FBt1/assignment_loss‚0.2805862160317335–ªHí“®‚(:ˆ ªŒåԉßµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.689' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.9378 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µê¯B=¯‚(:² ªŒåÔ¡‡· disk.drbd2266.in‚0 disk.rbd1.out‚ 0.919921875! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚12467.66259765625 disk.sda3.out‚67.5 proc.memory.rssMB‚ 3035.4375* proc.memory.percent‚18.526840209960938( disk./.usagePercent‚18.1716185623731 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 162.1796875 disk.drbd1716.out‚0% proc.memory.availableMB‚ 8.3046875 cpu‚25.212451080024067 network.sent‚ 3.37078e+06 network.recv‚688708 disk.rbd0.in‚0# disk.dm-0.out‚12467.66259765625 disk.drbd2080.in‚0 disk.drbd2265.in‚0$ memory_percent‚99.94931221008301 disk.dm-1.out‚67.5 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0% disk./.usageGB‚270.77024841308594 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚12467.66259765625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚67.5͉œ°‚(:‘ ¹ŒåÔÿ­ý´ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚301.22' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.24400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ŽQ±‚(:Æ ¹ŒåÔöº· disk.dm-0.in‚ 162.1796875 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.out‚ 68.0078125 proc.memory.rssMB‚ 3035.4375 cpu‚25.21114930839758% disk./.usageGB‚270.83888244628906( disk.nvme0c0n1.out‚12523.15966796875 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚99.94931221008301 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 68.0078125 disk.drbd2265.out‚0 disk.rbd0.out‚0* proc.memory.percent‚18.526840209960938# disk.dm-0.out‚12523.15966796875 disk.rbd0.in‚0& disk.nvme0n1.out‚12523.15966796875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 network.recv‚695484 disk.sda3.out‚ 68.0078125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 network.sent‚ 3.479668e+06 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.176224649929154! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2266.out‚0% proc.memory.availableMB‚ 8.3046875Ì Œ´“²‚(:ˆ ÈŒåÔ‚­îµ gpu.0.gpu‚65.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚261.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.2618 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØW S³‚(:È ÈŒåÔ¤Òݸ network.recv‚699101% disk./.usageGB‚270.86109924316406 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 162.1796875$ proc.memory.availableMB‚8.296875 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚12573.57958984375# proc.memory.rssMB‚ 3035.44140625* proc.memory.percent‚18.526864051818848 disk.sda2.in‚0# disk.dm-0.out‚12573.57958984375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0$ memory_percent‚99.94935989379883 disk.dm-0.in‚ 162.1796875 disk.sda1.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚12573.57958984375 disk.dm-1.out‚ 68.3828125 disk.rbd1.out‚ 0.919921875 disk.sda3.in‚0 disk.sda3.out‚ 68.3828125 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.sent‚ 3.485002e+06) disk./.usagePercent‚18.17771563780857 disk.rbd1.in‚0 disk.sda.out‚ 68.3828125 cpu‚25.20984223549154ÿ¼ò2´- _wandbruntime‚1886  _runtime‚1886‰‰€ÉµÃ _runtime‚1886.710983838 _step‚151 ' lr-AdamW/pg1‚0.00020254737793679877 ' lr-AdamW/pg2‚0.00020254737793679877 trainer/global_step‚1899 % _timestamp‚1.7884299083559406e+09¢¯Î¶È ' lr-AdamW/pg1‚0.00020254737793679877 ' lr-AdamW/pg2‚0.00020254737793679877 trainer/global_step‚1899 % _timestamp‚1.7884299083559406e+09 _runtime‚1886.710983838 _step‚151—‹g%¸2·- _wandbruntime‚1886  _runtime‚1886»6Þìù'¸óO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4043445859328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255524606420992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20093562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14760034 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74380132 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884299083567243e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40699941 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259568025888768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224008588025856 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _step‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74357022 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40676837 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255524580029440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4046551251968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255524580029440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33680191 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4046551251968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40676837 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4036625897091 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33680185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33680185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251246627228672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33680191 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20093565 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259571157672960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0!Ò¾ÿ̹øO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4046551251968 E?DeviceStatsMonitor.on_tra; ä¯2'in_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255524580029440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33680191 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4046551251968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40676837 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4036625897091 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33680185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33680185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251246627228672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33680191 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20093565 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259571157672960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4043445859328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255524606420992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20093562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14760034 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74380132 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884299083567243e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40699941 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259568025888768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224008588025856 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _step‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74357022 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40676837 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255524580029440 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251376049657856 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40721057 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259704764088320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4048663579136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5629983437824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34871471 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74395396 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4045558173184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224124022974464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4041838159315 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255656074117632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14767653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74418532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33697475 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14767137 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218493982437376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4048663579136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚1887.608534577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251376024229888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚1899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255656100509184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34870952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259701632290816 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20103815 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224123937740800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255414758920079 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 $ _timestamp‚1.788429909252778e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4045558173184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74418532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255656100509184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33697475 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251376049657856 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40721057 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259704764088320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4048663579136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5629983437824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34871471 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74395396 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4045558173184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224124022974464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4041838159315 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255656074117632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14767653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74418532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33697475 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14767137 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218493982437376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4048663579136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚1887.608534577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251376024229888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚1899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255656100509184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34870952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259701632290816 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20103815 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224123937740800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255414758920079 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 $ _timestamp‚1.788429909252778e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4045558173184™"YcÎ2½- _wandbruntime‚1887  _runtime‚1887m£ê©Y¾Ó /loss/FRt3/assignment_loss‚2.978294008411467 1loss/SRqqt2/assignment_loss‚2.918869060277939 /loss/FRt1/detection_loss‚2.2972573913309886 /loss/FRt2/detection_loss‚2.6954721212387085 /loss/FBt1/detection_loss‚1.3029000190860134 /loss/FRt1/assignment_loss‚3.346493548673132 0loss/FBt2/assignment_loss‚0.4523498900234699 1loss/SRqqt3/detection_loss‚1.2469718197711033 epoch‚1 % _timestamp‚1.7884299092546997e+09 /loss/FRt4/assignment_loss‚5.757623873258892 _runtime‚ 1887.60974929 2loss/SRqqt4/assignment_loss‚2.5425827564864325 1loss/SRqqt4/detection_loss‚2.5476787871327895 trainer/global_step‚1899 _step‚154 1loss/SRqqt1/detection_loss‚1.9161086905063416 /loss/FBt3/detection_loss‚2.9573643878102303 /loss/FRt2/assignment_loss‚4.974986271424727 2loss/SRqqt3/assignment_loss‚1.3678887370734079 1loss/FBt1/assignment_loss‚0.22367067870615978 1loss/FBt3/assignment_loss‚0.30093205397133715 0loss/FBt4/assignment_loss‚0.3277038544779841 /loss/FBt4/detection_loss‚1.5529193262600043 2loss/SRqqt1/assignment_loss‚1.6706376063580415 .loss/FRt3/detection_loss‚2.170702378731221 /loss/FBt2/detection_loss‚3.4141342639923096 .loss/FRt4/detection_loss‚3.225198419470536 0loss/SRqqt2/detection_loss‚2.434991753101349 &loss/total_loss‚54.623731697574584ê ­ÝÕ¿Ø 2loss/SRqqt4/assignment_loss‚2.5425827564864325 1loss/SRqqt4/detection_loss‚2.5476787871327895 trainer/global_step‚1899 _step‚154 1loss/SRqqt1/detection_loss‚1.9161086905063416 /loss/FBt3/detection_loss‚2.9573643878102303 /loss/FRt2/assignment_loss‚4.974986271424727 2loss/SRqqt3/assignment_loss‚1.3678887370734079 1loss/FBt1/assignment_loss‚0.22367067870615978 1loss/FBt3/assignment_loss‚0.30093205397133715 0loss/FBt4/assignment_loss‚0.3277038544779841 /loss/FBt4/detection_loss‚1.5529193262600043 2loss/SRqqt1/assignment_loss‚1.6706376063580415 .loss/FRt3/detection_loss‚2.170702378731221 /loss/FBt2/detection_loss‚3.4141342639923096 .loss/FRt4/detection_loss‚3.225198419470536 0loss/SRqqt2/detection_loss‚2.434991753101349 &loss/total_loss‚54.623731697574584 /loss/FRt3/assignment_loss‚2.978294008411467 1loss/SRqqt2/assignment_loss‚2.918869060277939 /loss/FRt1/detection_loss‚2.2972573913309886 /loss/FRt2/detection_loss‚2.6954721212387085 /loss/FBt1/detection_loss‚1.3029000190860134 /loss/FRt1/assignment_loss‚3.346493548673132 0loss/FBt2/assignment_loss‚0.4523498900234699 1loss/SRqqt3/detection_loss‚1.2469718197711(¥¨*‰033 epoch‚1 % _timestamp‚1.7884299092546997e+09 /loss/FRt4/assignment_loss‚5.757623873258892 _runtime‚ 1887.60974929šñ—\T“À‚(:ˆ ׌åÔòŽ”µ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”W…ÃYÁ‚(:Î ׌åÔô·ä¶ network.recv‚703791 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚12628.46337890625 disk.sda1.in‚0 disk.drbd1716.in‚0* disk./.usagePercent‚18.183046738701588 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 3035.44140625 disk.dm-1.out‚ 69.33984375 disk.sda3.out‚ 69.33984375 disk.sda.out‚ 69.33984375 disk.drbd2265.out‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 8.22265625& disk.nvme0n1.out‚12628.46337890625 disk.drbd2080.out‚0 disk.drbd2265.in‚0$ memory_percent‚99.94981288909912 cpu‚25.20864384705748 network.sent‚ 3.490865e+06% disk./.usageGB‚270.94053649902344 disk.rbd1.out‚ 1.068359375! disk.nvme0c0n1.in‚ 162.1796875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 162.1796875# disk.dm-0.out‚12628.46337890625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚18.526864051818848 proc.cpu.threads‚17_ô‚(:’ æŒåÔÐò—µ gpu.0.gpu‚53.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚309.841' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚61.96820000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\S"ùZÂ(:Ï æŒåԲ˷ disk.rbd1.in‚0 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 network.recv‚710437 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 cpu‚25.207402225421383 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 1.068359375$ memory_percent‚99.94823932647705* proc.memory.percent‚18.526864051818848 disk.sda1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚12678.29150390625# disk.dm-0.out‚12678.29150390625& proc.memory.availableMB‚ 8.48046875# proc.memory.rssMB‚ 3035.44140625 disk.sda.out‚ 69.80078125 disk.sda2.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 162.1796875 network.sent‚ 3.599588e+06 disk.drbd2266.out‚0* disk./.usagePercent‚18.185671071935086% disk./.usageGB‚270.97964096069336 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚12678.29150390625 disk.sda3.out‚ 69.80078125 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.out‚ 69.80078125 disk.drbd1716.in‚0Úpåb“Ä‚(:ˆ õŒåÔÊ͵ gpu.0.gpu‚76.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<üGTÅ‚(:É õŒåÔÃÀÖ· disk.rbd0.out‚0( disk.nvme0c0n1.out‚12731.88916015625 disk.sda1.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚18.195589315826624 disk.rbd0.in‚0& proc.memory.availableMB‚ 8.22265625 disk.dm-1.out‚ 70.4609375 disk.drbd2265.in‚0$ disk./.usageGB‚271.1274299621582 disk.drbd2266.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.sda3.out‚ 70.4609375 cpu‚25.206045594952776 proc.cpu.threads‚17 disk.sda.out‚ 70.4609375 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.in‚0# disk.dm-0.out‚12731.88916015625 disk.drbd1716.out‚0# proc.memory.rssMB‚ 3035.44140625 network.sent‚ 3.606e+06 network.recv‚715504 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0$ memory_percent‚99.94981288909912* proc.memory.percent‚18.526864051818848 disk.rbd1.in‚0 disk.rbd1.out‚ 1.1005859375& disk.nvme0n1.out‚12731.88916015625 disk.sda1.out‚0P)²º2Æ- _wandbruntime‚1933  _runtime‚1933”XŽ,ÉÇà ' lr-AdamW/pg1‚0.00020224471572508073 ' lr-AdamW/pg2‚0.00020224471572508073 trainer/global_step‚1949 % _timestamp‚1.7884299556156816e+09 _runtime‚1933.970758049 _step‚155”¥/ÎÈÈ _runtime‚1933.970758049 _step‚155 ' lr-AdamW/pg1‚0.00020224471572508073 ' lr-AdamW/pg2‚0.00020224471572508073 trainer/global_step‚1949 % _timestamp‚1.7884299556156816e+09›µ…?2É- _wandbruntime‚1933  _runtime‚1933‡™žú'ÊôO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266248360404992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41757793 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 trainer/global_step‚1949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262099284439040 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261858556432771 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149075965952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4149075965952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34544385 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚262099310830592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚262099310830592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41734689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5773767861248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257716477289472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚1933.971726608 _step‚156 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76279074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41757793 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4152181358592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20606212 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257716502717440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚266251492189184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229775854506496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224002086645248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142079143299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34544391 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224002114779648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 % _timestamp‚1.7884299556164334e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35748124 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76302184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262099284439040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4152181358592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4145182612215 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35747577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20606215 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41734689 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5773824973824 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35747577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20606215 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41734689 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5773824973824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266248360404992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41757793 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 trainer/global_step‚1949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262099284439040 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261858556432771 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149075965952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4149075965952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34544385 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚262099310830592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚262099310830592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41734689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5773767861248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257716477289472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚1933.971726608 _step‚156 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76279074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41757793 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4152181358592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20606212 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257716502717440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚266251492189184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229775854506496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224002086645248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142079143299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34544391 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224002114779648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 % _timestamp‚1.7884299556164334e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35748124 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76302184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262099284439040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928œ/´¡2Ì- _wandbruntime‚1934  _runtime‚1934™[ÙÕÍþM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266385098604544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚514 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41755779 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4147291405523 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34561669 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76317448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266385098604544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262230778527232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224114525643776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35765242 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4151188279808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257845899718656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚262230778527232 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262230804918784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257845874290688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266381966807040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76340584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15148777 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224114497509376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34561675 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4154293685760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34561675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41778909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15149291 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229891188918784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41755779 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚262230804918784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144187936399 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20616465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229891274152448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4154293685760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34561675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41778909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15149291 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229891188918784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41755779 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚262230804918784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144187936399 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20616465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229891274152448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34561669 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76317448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266385098604544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262230778527232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224114525643776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35765242 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4151188279808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257845899718656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚262230778527232 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262230804918784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257845874290688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266381966807040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76340584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15148777 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224114497509376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34561675 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517mX-ë2Ï- _wandbruntime‚1934  _runtime‚1934 M¹ÎUÐÏ /loss/FRt2/detection_loss‚2.8226016600926718 2loss/SRqqt3/assignment_loss‚1.4869467792495226 1loss/SRqqt4/assignment_loss‚2.073317607243856 /loss/FRt4/detection_loss‚3.4085108978407725 _step‚158 0loss/FRt3/assignment_loss‚2.6104483675148527 1loss/FBt2/assignment_loss‚0.10403716553988715 0loss/SRqqt4/detection_loss‚2.569459206528134 epoch‚1 /loss/FRt1/detection_loss‚2.1735421346158397 1loss/SRqqt1/detection_loss‚1.4015628787289773 /loss/SRqqt2/detection_loss‚2.33381386236711 /loss/FBt1/detection_loss‚1.4370742758115134 %loss/total_loss‚52.59667507959314 0loss/FRt1/assignment_loss‚2.5006669565122954 /loss/FRt4/assignment_loss‚4.884846653257098 1loss/SRqqt1/assignment_loss‚1.815042081388778 0loss/FBt3/assignment_loss‚0.4137361630797386 /loss/FRt2/assignment_loss‚5.092826914787293 1loss/FBt1/assignment_loss‚0.19382299390093308 .loss/FRt3/detection_loss‚2.000303434618449 .loss/FBt3/detection_loss‚2.936279684305191 /loss/FBt4/detection_loss‚1.8839257195603718 % _timestamp‚1.7884299565174098e+09 0loss/SRqqt2/assignment_loss‚2.36750039638895 1loss/SRqqt3/detection_loss‚1.2441673417790577 .loss/FBt2/detection_loss‚4.459622406959534 trainer/global_step‚1949 _runtime‚1934.872404034 0loss/FBt4/assignment_loss‚0.3826194975223208ßôæ ZÑÔ /loss/FRt1/detection_loss‚2.1735421346158397 1loss/SRqqt1/detection_loss‚1.4015628787289773 /loss/SRqqt2/detection_loss‚2.33381386236711 /loss/FBt1/detection_loss‚1.4370742758115134 %loss/total_loss‚52.59667507959314 0loss/FRt1/assignment_loss‚2.5006669565122954 /loss/FRt4/assignment_loss‚4.884846653257098 1loss/SRqqt1/assignment_loss‚1.815042081388778 0loss/FBt3/assignment_loss‚0.4137361630797386 /loss/FRt2/assignment_loss‚5.092826914787293 1loss/FBt1/assignment_loss‚0.19382299390093308 .loss/FRt3/detection_loss‚2.000303434618449 .loss/FBt3/detection_loss‚2.936279684305191 /loss/FBt4/detection_loss‚1.8839257195603718 % _timestamp‚1.7884299565174098e+09 0loss/SRqqt2/assignment_loss‚2.36750039638895 1loss/SRqqt3/detection_loss‚1.2441673417790577 .loss/FBt2/detection_loss‚4.459622406959534 trainer/global_step‚1949 _runtime‚1934.872404034 0loss/FBt4/assignment_loss‚0.3826194975223208 /loss/FRt2/detection_loss‚2.8226016600926718 2loss/SRqqt3/assignment_loss‚1.4869467792495226 1loss/SRqqt4/assignment_loss‚2.073317607243856 /loss/FRt4/detection_loss‚3.4085108978407725 _step‚158 0loss/FRt3/assignment_loss‚2.6104483675148527 1loss/FBt2/assignment_loss‚0.10403716553988715 0loss/SRqqt4/detection_loss‚2.569459206528134 epoch‚1žz@o¥žÒ‚(:“ „åÔè€øµ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚295.847' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.169399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p¸'}WÓ‚(:Ì „åÔϽó· disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2080.in‚0 disk.drbd2080.out‚0% disk./.usageGB‚271.19549560546875 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0 cpu‚25.204845736257504 disk.sda3.out‚ 70.88671875 network.sent‚ 3.611263e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0 disk.sda.out‚ 70.88671875* proc.memory.percent‚18.526864051818848) disk./.usagePercent‚18.20015725826007 disk.drbd1716.out‚0# proc.memory.availableMB‚7.96875 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.1005859375# disk.dm-0.out‚12790.37353515625$ memory_percent‚99.95136260986328 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda1.in‚0 network.recv‚718865( disk.nvme0c0n1.out‚12790.37353515625 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0# proc.memory.rssMB‚ 3035.44140625 disk.drbd2265.out‚0 disk.dm-1.out‚ 70.88671875 proc.cpu.threads‚17& disk.nvme0n1.out‚12790.37353515625 disk.drbd2266.in‚0ÃÆVx“Ô‚(:ˆ “åÔÒ¡«µ gpu.0.gpu‚83.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚291.244' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éšò6VÕ‚(:Ë “åÔùýµ· disk.sda2.in‚0 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.drbd2080.in‚0$ disk./.usageGB‚271.2037887573242 disk.rbd1.out‚ 1.16357421875( disk.nvme0c0n1.out‚12859.07861328125 disk.drbd1716.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 3035.44140625 disk.drbd2266.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.sent‚ 3.719863e+06) disk./.usagePercent‚18.20071381863951* proc.memory.percent‚18.526864051818848 disk.nvme0n1.in‚ 162.1796875 disk.rbd1.in‚0 cpu‚25.20355636942991& disk.nvme0n1.out‚12859.07861328125 disk.dm-1.in‚0# disk.dm-0.out‚12859.07861328125 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ proc.memory.availableMB‚7.921875 disk.dm-1.out‚ 71.56640625 disk.sda.out‚ 71.56640625 network.recv‚725259 disk.sda2.out‚0 disk.sda3.out‚ 71.56640625 disk.sda.in‚0# memory_percent‚99.9516487121582 proc.cpu.threads‚17:”žÖ‚(:’ ¢åÔ¿¸µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.112' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.22240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y ZYׂ(:Î ¢åÔ´œó¶ cpu‚25.20241277399833 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.16357421875# proc.memory.rssMB‚ 3035.44140625 proc.cpu.threads‚17& disk.nvme0n1.out‚12912.13427734375 disk.dm-1.out‚ 71.96484375! disk.nvme0c0n1.in‚ 162.1796875( disk.nvme0c0n1.out‚12912.13427734375 disk.sda3.out‚ 71.96484375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 network.sent‚ 3.72475e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.sda.out‚ 71.96484375$ memory_percent‚99.95009899139404% disk./.usageGB‚271.33352279663086# disk.dm-0.out‚12912.13427734375 disk.drbd2080.out‚0 network.recv‚728589) disk./.usagePercent‚18.20942037887149 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0& proc.memory.availableMB‚ 8.17578125* proc.memory.percent‚18.526864051818848Xg˜“Ø‚(:ˆ ±åÔ ùµ gpu.0.gpu‚49.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òä­¼NÙ‚(:à ±åÔÄÀï¶ disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.2265625( disk.nvme0c0n1.out‚12975.81787109375 cpu‚25.20127382606365$ disk./.usageGB‚271.3836326599121 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.in‚0# disk.dm-0.out‚12975.81787109375 disk.sda3.out‚ 72.484375 disk.rbd0.out‚0$ memory_percent‚99.95007514953613 disk.sda1.out‚0% proc.memory.availableMB‚ 8.1796875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd2080.out‚0 network.sent‚ 3.731016e+06) disk./.usagePercent‚18.21278329384127& disk.nvme0n1.out‚12975.81787109375 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 162.1796875# proc.memory.rssMB‚ 3035.44140625 disk.sda2.in‚0 disk.dm-1.out‚ 72.484375 disk.dm-0.in‚ 162.1796875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.out‚ 72.484375 network.recv‚733301* proc.memory.percent‚18.526864051818848 disk.drbd2266.out‚0 disk.drbd2265.in‚0BkÁJ2Ú- _wandbruntime‚1981  _runtime‚1981ÖßÀ'ÈÛ ' lr-AdamW/pg2‚0.00020192122210288047 trainer/global_step‚1999 % _timestamp‚1.7884300027783551e+09 _runtime‚ 1981.13349826 _step‚159 ' lr-AdamW/pg1‚0.00020192122210288047T{þáÍÜÇ trainer/global_step‚1999 % _timestamp‚1.7884300027783551e+09 _runtime‚ 1981.13349826 _step‚159 ' lr-AdamW/pg1‚0.00020192122210288047 ' lr-AdamW/pg2‚0.00020192122210288047Ÿ¹ª«P2Ý- _wandbruntime‚1981  _runtime‚1981‰+s´ù'ÞóO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272931826705408 trainer/global_step‚1999 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268433859739779 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78224236 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21118862 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268673988848640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235542747314688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268674015240192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4257811465216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42792541 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42792541 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4257811465216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272928694921216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42815645 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42815645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15519004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229622657986048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268674015240192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36637869 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264186327350272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272931826705408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36637326 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272928694921216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21118865 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788430002779238e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268674015240192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4257811465216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42792541 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42792541 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4257811465216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272928694921216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42815645 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42815645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15519004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229622657986048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268674015240192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36637869 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264186327350272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272931826705408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36637326 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272928694921216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21118865 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788430002779238e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272931826705408 trainer/global_step‚1999 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268433859739779  öj2à- _wandbruntime‚1982  _runtime‚1982€bo…'áþM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264315724351488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229735068850176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78239500 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268565365534095 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78262636 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273062301323264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268805482936832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1999 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268805509328384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235658009420800 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15526011 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229735040715776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35425875 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42836761 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36655659 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268805509328384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36655126 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 _runtime‚1982.040012166 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235658094654464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884300036842573e+09 _step‚161 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42813631 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21129118 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4249641182607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42813631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35425869 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78262636 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78239500 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4256818386432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42836761 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273065433120768 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268568494431187 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268805482936832 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42836761 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36655659 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268805509328384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36655126 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 _runtime‚1982.040012166 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235658094654464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884300036842573e+09 _step‚161 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42813631 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21129118 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4249641182607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42813631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35425869 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78262636 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78239500 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4256818386432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42836761 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273065433120768 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268568494431187 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268805482936832 F2ì- _wandbruntime‚2027  _runtime‚2027+ƒâ¾Èí ' lr-AdamW/pg1‚0.00020157696448578399 ' lr-AdamW/pg2‚0.00020157696448578399 trainer/global_step‚2049 $ _timestamp‚1.788430049538897e+09 _runtime‚2027.893912956 _step‚163‘¤š1ÍîÇ ' lr-AdamW/pg1‚0.00020157696448578399 ' lr-AdamW/pg2‚0.00020157696448578399 trainer/global_step‚2049 $ _timestamp‚1.788430049538897e+09 _runtime‚2027.893912956 _step‚163£姨t2ï- _wandbruntime‚2027  _runtime‚2027ýÊg–ú'ðôO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279612161221632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279612161221632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43850393 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21631515 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80146288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241309787821056 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4352985635715 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚275248719649792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235243173058048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275009163046787 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37531014 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4363441571840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21631512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6066586628608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36272791 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36272785 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36272791 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279609029437440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241309700476928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279609029437440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37530456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15898944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 % _timestamp‚1.7884300495398178e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4360336179200 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43873497 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15899499 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 _runtime‚2027.895183093 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275012291943671 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270656202839040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235243201192448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36272785 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6066527418880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270656177411072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚2049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4363441571840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4356089104631 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚275248693258240 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚275248693258240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4363441571840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21631512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6066586628608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36272791 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36272785 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36272791 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279609029437440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241309700476928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279609029437440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37530456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15898944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 % _timestamp‚1.7884300495398178e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4360336179200 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43873497 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15899499 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 _runtime‚2027.895183093 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275012291943671 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270656202839040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235243201192448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36272785 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6066527418880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270656177411072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚2049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4363441571840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4356089104631 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚275248693258240 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚275248693258240 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 2028.78719374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚275380187346432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4365553899008 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚275380187346432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4362448493056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚275380213737984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚275380213737984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279745767636992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4362448493056 _step‚165 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36290075 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279745767636992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6069452937728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15906629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241425036859904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21641765 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80184688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36290075 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275140668841103 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80161552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21641768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37548394 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884300504313664e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36290069 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537i5|@'ô‚N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43871483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15907166 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43871483 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6069510036992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241425122093568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 2028.78719374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚275380187346432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4365553899008 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚275380187346432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4362448493056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚275380213737984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚275380213737984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279745767636992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4362448493056 _step‚165 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36290075 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279745767636992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6069452937728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15906629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241425036859904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21641765 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80184688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36290075 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275140668841103 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80161552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21641768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37548394 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884300504313664e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36290069 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275143797738195 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4358197897939 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456¥Q#d2õ- _wandbruntime‚2028  _runtime‚2028ö_BXöÒ 0loss/FBt4/assignment_loss‚0.5439445089564708 /loss/FRt3/detection_loss‚2.5178536540124474 0loss/SRqqt4/detection_loss‚3.189623638987541 1loss/FBt1/assignment_loss‚0.30467756840166027 /loss/FRt2/detection_loss‚2.5765908862415112 /loss/FBt1/detection_loss‚1.4785890105801325 .loss/FRt1/assignment_loss‚2.77738019824028 .loss/FBt2/detection_loss‚3.184518575668335 epoch‚1 _step‚166 2loss/SRqqt4/assignment_loss‚2.0193643867969513 0loss/FBt2/assignment_loss‚0.3977758669320287 1loss/SRqqt1/detection_loss‚1.4479530782749255 /loss/FBt3/detection_loss‚3.6492778956890106 trainer/global_step‚2049 /loss/FRt3/assignment_loss‚2.064901971235508 2loss/SRqqt1/assignment_loss‚1.8477361963854895 1loss/SRqqt2/assignment_loss‚2.288260225703319 /loss/FRt1/detection_loss‚2.0082233366039066 _runtime‚2028.788321448 % _timestamp‚1.7884300504333565e+09 /loss/FRt4/assignment_loss‚5.495028789838155 .loss/FRt4/detection_loss‚2.789268304904302 1loss/SRqqt2/detection_loss‚2.4547623217105867 /loss/FBt4/detection_loss‚1.6001626754659222 %loss/total_loss‚52.63311253000708 1loss/SRqqt3/detection_loss‚1.1751624986875888 /loss/FRt2/assignment_loss‚5.263555514185052 2loss/SRqqt3/assignment_loss‚1.4323623208978842 1loss/FBt3/assignment_loss‚0.12613910560806593O†$]÷× trainer/global_step‚2049 /loss/FRt3/assignment_loss‚2.064901971235508 2loss/SRqqt1/assignment_loss‚1.8477361963854895 1loss/SRqqt2/assignment_loss‚2.288260225703319 /loss/FRt1/detection_loss‚2.0082233366039066 _runtime‚2028.788321448 % _timestamp‚1.7884300504333565e+09 /loss/FRt4/assignment_loss‚5.495028789838155 .loss/FRt4/detection_loss‚2.789268304904302 1loss/SRqqt2/detection_loss‚2.4547623217105867 /loss/FBt4/detection_loss‚1.6001626754659222 %loss/total_loss‚52.63311253000708 1loss/SRqqt3/detection_loss‚1.1751624986875888 /loss/FRt2/assignment_loss‚5.263555514185052 2loss/SRqqt3/assignment_loss‚1.4323623208978842 1loss/FBt3/assignment_loss‚0.12613910560806593 0loss/FBt4/assignment_loss‚0.5439445089564708 /loss/FRt3/detection_loss‚2.5178536540124474 0loss/SRqqt4/detection_loss‚3.189623638987541 1loss/FBt1/assignment_loss‚0.30467756840166027 /loss/FRt2/detection_loss‚2.5765908862415112 /loss/FBt1/detection_loss‚1.4785890105801325 .loss/FRt1/assignment_loss‚2.77738019824028 .loss/FBt2/detection_loss‚3.184518575668335 epoch‚1 _step‚166 2loss/SRqqt4/assignment_loss‚2.0193643867969513 0loss/FBt2/assignment_loss‚0.3977758669320287 1loss/SRqqt1/detection_loss‚1.4479530782749255 /loss/FBt3/detection_loss‚3.6492778956890106¦i†˜@“ø‚(:ˆ íåÔ•¨‘µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚283.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÿ&`Vù‚(:Ë íåÔ€èâ¶# disk.dm-0.out‚13201.79248046875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.in‚0$ memory_percent‚99.95074272155762 disk.sda.out‚ 74.64453125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 74.64453125% proc.memory.availableMB‚ 8.0703125 proc.cpu.threads‚17 disk.drbd2265.in‚0& disk.nvme0n1.out‚13201.79248046875* proc.memory.percent‚18.526864051818848 disk.drbd2265.out‚0% disk./.usageGB‚271.54985427856445 disk.dm-1.out‚ 74.64453125 disk.rbd1.out‚ 1.2265625! disk.nvme0c0n1.in‚ 162.1796875( disk.nvme0c0n1.out‚13201.79248046875 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 3.857009e+06* disk./.usagePercent‚18.223938566138255 disk.drbd2266.out‚0 disk.dm-0.in‚ 162.1796875 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3035.44140625 network.recv‚752125 cpu‚25.19664053046673 disk.dm-1.in‚0ahÊÑžú‚(:“ üåÔ÷‘µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.684999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐWRû‚(:Ç üåÔ¤½å¶ disk.sda1.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 74.98828125 network.recv‚757889 disk.sda3.out‚ 74.98828125! disk.nvme0c0n1.in‚ 162.1796875" disk.dm-0.out‚13252.0205078125$ memory_percent‚99.95081424713135 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚13252.0205078125# proc.memory.rssMB‚ 3035.44140625 cpu‚25.195702940849056 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0& proc.memory.availableMB‚ 8.05859375$ disk./.usageGB‚271.5325126647949 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0 disk.rbd1.out‚ 1.2265625 network.sent‚ 3.964862e+06( disk./.usagePercent‚18.2227747558876 disk.drbd1716.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚13252.0205078125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 74.98828125 disk.drbd2265.out‚0 disk.dm-0.in‚ 162.1796875* proc.memory.percent‚18.526864051818848CŽ²£‘ü‚(:† ‹ŽåÔוµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èü¿¹Dý‚(:¹ ‹ŽåÔøî¶ disk.drbd2080.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.2265625 network.recv‚762546$ disk./.usageGB‚271.5707588195801! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.sda.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 network.sent‚ 3.971058e+06 disk.dm-1.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚13314.6005859375 disk.sda.out‚75.625 disk.drbd2265.in‚0% disk.nvme0n1.out‚13314.6005859375 disk.dm-1.out‚75.625 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0& proc.memory.availableMB‚ 7.55078125# proc.memory.rssMB‚ 3035.44140625 cpu‚25.194658236915263$ memory_percent‚99.95391368865967 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0" disk.dm-0.out‚13314.6005859375 disk.sda3.in‚0 disk.sda3.out‚75.625 disk.sda1.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚18.22534148742589 disk.sda2.in‚0* proc.memory.percent‚18.526864051818848›;»¥2þ- _wandbruntime‚2074  _runtime‚2074ž¢ªZÇÿÁ % _timestamp‚1.7884300966409044e+09 _runtime‚2074.995944376 _step‚167 & lr-AdamW/pg1‚0.0002012120146165627 & lr-AdamW/pg2‚0.0002012120146165627 trainer/global_step‚2099£¢ÖÂÌ€Æ & lr-AdamW/pg1‚0.0002012120146165627 & lr-AdamW/pg2‚0.0002012120146165627 trainer/global_step‚2099 % _timestamp‚1.7884300966409044e+09 _runtime‚2074.995944376 _step‚167§x©üˆ2- _wandbruntime‚2074  _runtime‚2074j¸b*ù'‚óO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37136991 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277126027471872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚281823397667840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44931349 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240863716264448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22144162 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461542350839 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37136991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286289363953664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4469071678464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37136985 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44908245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚2074.996892916 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4469071678464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247076830614016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281587595250679 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22144165 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 $ _timestamp‚1.788430096641617e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38424838 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚281823424059392 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286292495737856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4458438881923 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4465966285824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38424309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6213169364992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44908245 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚281823397667840 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44931349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16280673 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277126052899840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _step‚168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286289363953664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16280147 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281584466353795 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286292495737856 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240863744398848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4465966285824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247076913763840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 trainer/global_step‚2099 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192ÞZ]•cƒøO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286292495737856 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240863744398848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4465966285824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247076913763840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 trainer/global_step‚2099 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 VGÐös*›"DeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37136991 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277126027471872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚281823397667840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44931349 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240863716264448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22144162 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461542350839 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37136991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286289363953664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4469071678464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37136985 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44908245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚2074.996892916 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4469071678464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247076830614016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281587595250679 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22144165 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55015424 $ _timestamp‚1.788430096641617e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38424838 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚281823424059392 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286292495737856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4458438881923 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4465966285824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38424309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6213169364992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44908245 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚281823397667840 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44931349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16280673 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277126052899840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _step‚168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286289363953664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16280147 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281584466353795¨4;7E2„- _wandbruntime‚2075  _runtime‚2075¨à4 '…þM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22154418 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82106740 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286426102153216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286422970355712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16288120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6216055920640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281954891756032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚281954918147584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82106740 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22154415 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44929335 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884300975315921e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82083604 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44952465 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38442035 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247192266185728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44929335 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4471184005632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37154269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38442538 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281715972148111 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286426102153216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440«° '†ƒN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44929335 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4471184005632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37154269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38442538 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281715972148111 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286426102153216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22154418 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82106740 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286426102153216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286422970355712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16288120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6216055920640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281954891756032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚281954918147584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82106740 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22154415 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44929335 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884300975315921e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82083604 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44952465 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38442035 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247192266185728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136©@L®2‡- _wandbruntime‚2075  _runtime‚2075•IXˆÒ /loss/FRt2/assignment_loss‚5.529985211112282 /loss/FRt1/assignment_loss‚2.936687234502572 2loss/SRqqt1/assignment_loss‚1.4894723324726025 _step‚170 0loss/FRt3/assignment_loss‚3.1834840214252473 /loss/FRt4/assignment_loss‚5.390992840131124 0loss/FBt2/assignment_loss‚0.5848106754478627 1loss/FBt4/assignment_loss‚0.45253100656343903 /loss/FRt1/detection_loss‚2.0888417603877873 /loss/FBt2/detection_loss‚2.3562155755666585 %loss/total_loss‚51.45828733044567 epoch‚1 1loss/FBt1/assignment_loss‚0.21394525469402087 .loss/FRt2/detection_loss‚2.608790791395939 1loss/SRqqt1/detection_loss‚1.5406369239091873 0loss/SRqqt3/detection_loss‚0.976291252944905 /loss/FBt4/detection_loss‚1.4691222436925284 trainer/global_step‚2099 1loss/FBt3/assignment_loss‚0.08948612463773316 /loss/FRt3/detection_loss‚2.1876864683628083 .loss/FRt4/detection_loss‚3.391866836282942 2loss/SRqqt3/assignment_loss‚1.3263026987506936 1loss/SRqqt4/assignment_loss‚2.250221401453018 0loss/SRqqt2/detection_loss‚2.929934642816845 /loss/FBt1/detection_loss‚1.2250260413485639 .loss/FBt3/detection_loss‚2.675460344552994 _runtime‚2075.888376653 % _timestamp‚1.7884300975334027e+09 1loss/SRqqt2/assignment_loss‚2.165507592652973 1loss/SRqqt4/detection_loss‚2.3949880553409457:Æ)]‰× 1loss/FBt3/assignment_loss‚0.08948612463773316 /loss/FRt3/detection_loss‚2.1876864683628083 .loss/FRt4/detection_loss‚3.391866836282942 2loss/SRqqt3/assignment_loss‚1.3263026987506936 1loss/SRqqt4/assignment_loss‚2.250221401453018 0loss/SRqqt2/detection_loss‚2.929934642816845 /loss/FBt1/detection_loss‚1.2250260413485639 .loss/FBt3/detection_loss‚2.675460344552994 _runtime‚2075.888376653 % _timestamp‚1.7884300975334027e+09 1loss/SRqqt2/assignment_loss‚2.165507592652973 1loss/SRqqt4/detection_loss‚2.3949880553409457 /loss/FRt2/assignment_loss‚5.529985211112282 /loss/FRt1/assignment_loss‚2.936687234502572 2loss/SRqqt1/assignment_loss‚1.4894723324726025 _step‚170 0loss/FRt3/assignment_loss‚3.1834840214252473 /loss/FRt4/assignment_loss‚5.390992840131124 0loss/FBt2/assignment_loss‚0.5848106754478627 1loss/FBt4/assignment_loss‚0.45253100656343903 /loss/FRt1/detection_loss‚2.0888417603877873 /loss/FBt2/detection_loss‚2.3562155755666585 %loss/total_loss‚51.45828733044567 epoch‚1 1loss/FBt1/assignment_loss‚0.21394525469402087 .loss/FRt2/detection_loss‚2.608790791395939 1loss/SRqqt1/detection_loss‚1.5406369239091873 0loss/SRqqt3/detection_loss‚0.976291252944905 /loss/FBt4/detection_loss‚1.4691222436925284 trainer/global_step‚2099ªbª’Š‚(:‡ šŽåÔšþ´ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚273.275' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.655 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äYEn‹‚(:É šŽåÔ™Ÿã¶ disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚13366.83837890625# disk.dm-0.out‚13366.83837890625* proc.memory.percent‚18.526864051818848% disk./.usageGB‚271.61231231689453 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.2265625! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd1716.out‚0) disk./.usagePercent‚18.22813017749637& disk.nvme0n1.out‚13366.83837890625 disk.sda3.out‚ 76.0078125 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 cpu‚25.193632578309163 disk.sda2.in‚0 disk.sda1.in‚LÚu=æ0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 network.sent‚ 3.976894e+06 disk.sda2.out‚0 disk.dm-0.in‚ 162.1796875 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0$ memory_percent‚99.95291233062744& proc.memory.availableMB‚ 7.71484375 disk.drbd2080.in‚0 disk.sda.out‚ 76.0078125# proc.memory.rssMB‚ 3035.44140625 proc.cpu.threads‚17 network.recv‚766766 disk.dm-1.out‚ 76.0078125 disk.sda3.in‚0 disk.sda.in‚0jê,=žŒ‚(:“ ©Žåԩ犵 gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.222' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.044399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 A‚H‚(:½ ©ŽåÔŒ‚è¶$ memory_percent‚99.95458126068115# proc.memory.rssMB‚ 3035.44140625$ disk./.usageGB‚271.6649589538574 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 7.44140625 network.recv‚773729 disk.rbd1.in‚0" disk.dm-0.out‚13430.8974609375 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 162.1796875' disk.nvme0c0n1.out‚13430.8505859375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚ 4.085645e+06 disk.sda3.in‚0 disk.sda.in‚0* disk./.usagePercent‚18.231663337476423 disk.dm-1.out‚76.5625 disk.drbd2266.in‚0 disk.dm-0.in‚ 162.1796875 disk.drbd2265.out‚0 disk.rbd0.out‚0 cpu‚25.192500418293996 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚76.5625 disk.rbd0.in‚0* proc.memory.percent‚18.526864051818848% disk.nvme0n1.out‚13430.8583984375 disk.rbd1.out‚ 1.2265625 disk.sda3.out‚76.5625 disk.drbd2265.in‚0ïŒâè“Ž‚(:ˆ ¸ŽåÔƃø¶ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.888' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.5776 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@/–T‚(:É ¸ŽåÔÛ³¥¹% disk./.usageGB‚271.71342849731445 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚99.95465278625488' disk.nvme0c0n1.out‚13462.9404296875 network.recv‚776993 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.sda3.in‚0 cpu‚25.191441176640947* disk./.usagePercent‚18.234916169206482 disk.sda.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 76.99609375 disk.dm-0.in‚ 162.1796875 network.sent‚ 4.090929e+06" disk.dm-0.out‚13462.9404296875* proc.memory.percent‚18.526864051818848 disk.drbd2080.in‚0 disk.sda.out‚ 76.99609375 disk.drbd2265.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚13462.9404296875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 76.99609375% proc.memory.availableMB‚ 7.4296875 proc.cpu.threads‚17 disk.rbd1.out‚ 1.2265625 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 3035.44140625E"2- _wandbruntime‚2121  _runtime‚2121²Pç¥Ç‘Á _step‚171 & lr-AdamW/pg1‚0.0002008264485502223 & lr-AdamW/pg2‚0.0002008264485502223 trainer/global_step‚2149 % _timestamp‚1.7884301435126717e+09 _runtime‚2121.867736886œäÅÌ’Æ % _timestamp‚1.7884301435126717e+09 _runtime‚2121.867736886 _step‚171 & lr-AdamW/pg1‚0.0002008264485502223 & lr-AdamW/pg2‚0.0002008264485502223 trainer/global_step‚2149«etÜh2“- _wandbruntime‚2121  _runtime‚2121´  Ÿù'”óO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4571596392448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288398102077440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4571596392448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16659422 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283595902960640 trainer/global_step‚2149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45989201 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45966097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38001191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252843609896448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39316237 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292969698469888 _runtime‚ 2121.86869232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38001185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288398102077440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚172 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288398128468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6359263081472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288159769660803 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39315668 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292972830254080 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246484287605248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6359322291200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4574701785088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22656812 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4566995597047 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38001185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4574701785088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292969698469888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440<4Ô[þ'•øO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38001185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288398102077440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚172 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288398128468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6359263081472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288159769660803 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39315668 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292972830254080 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246484287605248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6359322291200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4574701785088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22656812 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4566995597047 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38001185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4574701785088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292969698469888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4571596392448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288398102077440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4571596392448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16659422 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283595902960640 trainer/global_step‚2149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45989201 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45966097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38001191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252843609896448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39316237 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292969698469888 _runtime‚ 2121.86869232 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39333230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288294404352211 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4569104390355 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 % _timestamp‚1.7884301444055526e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576814112256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84028792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4576814112256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39333764 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38018475 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283725274533888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46010317 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45987187 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45987187 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293106436669440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573708706304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22667065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59196416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22667068 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2122.761226374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288529622557184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4566000921231 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚173 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288529596165632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84028792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16666696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84005656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16666165 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38018469 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878¯ '˜ƒN T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293106436669440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38018475 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84005656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6362189284352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39333230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288294404352211 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4569104390355 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 % _timestamp‚1.7884301444055526e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576814112256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84028792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4576814112256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39333764 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38018475 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283725274533888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46010317 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45987187 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45987187 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293106436669440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573708706304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22667065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59196416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22667068 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2122.761226374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288529622557184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4566000921231 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚173 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288529596165632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84028792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16666696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84005656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16666165 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38018469 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293103304871936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288291275455119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46010317 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294738201864695 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294972832878592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23169465 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258610428092928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258610340748800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252104830811648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299650032986112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17037991 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294972806487040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17038540 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299653164770304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38865385 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4680331891712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _runtime‚2168.640609177 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6505597281280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47023949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4680331891712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85912444 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40208005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4677226499072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290065727593472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294972806487040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47047053 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 …îŒÉ §øO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299653164770304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47023949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23169462 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294738201864695 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294972832878592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23169465 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258610428092928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258610340748800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252104830811648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299650032986112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17037991 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294972806487040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17038540 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299653164770304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38865385 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4680331891712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _runtime‚2168.640609177 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6505597281280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47023949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4680331891712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85912444 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40208005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4677226499072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290065727593472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294972806487040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47047053 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38865391 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299650032986112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47047053 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4669345374339 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252104802677248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6505538071552 _step‚176 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 $ _timestamp‚1.788430190285234e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294972832878592 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4672448843255 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚2199 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85950844 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85927708 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6508453198848 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85950844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295104326966784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295104300575232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38882669 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4674557636563 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40225750 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290195150022656 trainer/global_step‚2199 _runtime‚2169.539307198 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47045039 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4682444218880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47068169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23179718 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294866578762127 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4679338812928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290195124594688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38882669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4679338812928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294869707659219 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17045489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38882675 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40225204 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258725666740224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59196416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47045039 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47068169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258725754071040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87330816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252217213541376 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85950844 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85927708 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6508453198848 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85950844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295104326966784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295104300575232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38882669 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4674557636563 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40225750 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290195150022656 trainer/global_step‚2199 _runtime‚2169.539307198 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47045039 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4682444218880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47068169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23179718 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294866578762127 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4679338812928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290195124594688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38882669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4679338812928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294869707659219 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17045489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38882675 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40225204 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258725666740224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59196416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47045039 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47068169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258725754071040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968±‚ä^2«- _wandbruntime‚2169  _runtime‚2169ˆNã‘W¬Ñ 2loss/SRqqt4/assignment_loss‚2.0967919492516023 .loss/FRt4/detection_loss‚3.285787366685413 1loss/FBt1/assignment_loss‚0.17535392603259733 .loss/FRt3/detection_loss‚2.107401269674301 /loss/FBt4/detection_loss‚1.3772891764115716 1loss/SRqqt1/detection_loss‚1.7269473761320113 trainer/global_step‚2199 /loss/FRt2/assignment_loss‚4.703487187623978 /loss/FRt4/assignment_loss‚5.415206750233968 2loss/SRqqt1/assignment_loss‚1.4660739892721175 1loss/FBt2/assignment_loss‚0.37784449233974254 1loss/SRqqt3/detection_loss‚1.3163451255265488 2loss/SRqqt3/assignment_loss‚1.2639916218951883 .loss/FRt2/detection_loss‚2.599400110542774 1loss/SRqqt4/detection_loss‚2.6024614593078352 -loss/FBt1/detection_loss‚1.23914778859665 /loss/FBt3/detection_loss‚2.4602576245864234 % _timestamp‚1.7884301911854873e+09 /loss/FRt1/assignment_loss‚2.523644427458445 0loss/FRt3/assignment_loss‚2.9414008617401124 0loss/FBt3/assignment_loss‚0.6644243694997082 .loss/FRt1/detection_loss‚2.038423368537968 -loss/FBt2/detection_loss‚3.21294412513574 epoch‚1 _runtime‚2169.540508936 &loss/total_loss‚50.093521674321025 2loss/SRqqt2/assignment_loss‚1.8438121457310284 0loss/FBt4/assignment_loss‚0.2732731944378817 1loss/SRqqt2/detection_loss‚2.3818119676674114 _step‚178ÌÍX\­Ö 2loss/SRqqt3/assignment_loss‚1.2639916218951883 .loss/FRt2/detection_loss‚2.599400110542774 1loss/SRqqt4/detection_loss‚2.6024614593078352 -loss/FBt1/detection_loss‚1.23914778859665 /loss/FBt3/detection_loss‚2.4602576245864234 % _timestamp‚1.7884301911854873e+09 /loss/FRt1/assignment_loss‚2.523644427458445 0loss/FRt3/assignment_loss‚2.9414008617401124 0loss/FBt3/assignment_loss‚0.6644243694997082 .loss/FRt1/detection_loss‚2.038423368537968 -loss/FBt2/detection_loss‚3.21294412513574 epoch‚1 _runtime‚2169.540508936 &loss/total_loss‚50.093521674321025 2loss/SRqqt2/assignment_loss‚1.8438121457310284 0loss/FBt4/assignment_loss‚0.2732731944378817 1loss/SRqqt2/detection_loss‚2.3818119676674114 _step‚178 2loss/SRqqt4/assignment_loss‚2.0967919492516023 .loss/FRt4/detection_loss‚3.285787366685413 1loss/FBt1/assignment_loss‚0.17535392603259733 .loss/FRt3/detection_loss‚2.107401269674301 /loss/FBt4/detection_loss‚1.3772891764115716 1loss/SRqqt1/detection_loss‚1.7269473761320113 trainer/global_step‚2199 /loss/FRt2/assignment_loss‚4.703487187623978 /loss/FRt4/assignment_loss‚5.415206750233968 2loss/SRqqt1/assignment_loss‚1.4660739892721175 1loss/FBt2/assignment_loss‚0.37784449233974254 1loss/SRqqt3/detection_loss‚1.3163451255265488²[úð®‚(:’ ôŽåÔªüŒµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚259.954' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.99080000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç„ÚùL¯‚(:Á ôŽåÔÃü׶ disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 1.2265625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚18.526864051818848 disk.rbd1.in‚0# disk.dm-0.out‚13671.35498046875 disk.sda3.out‚79.09375 cpu‚25.18741986872923 proc.cpu.threads‚17 network.sent‚ 4.216782e+06% disk./.usageGB‚271.72875213623047 disk.dm-1.out‚79.09375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd1716.in‚0 disk.sda.out‚79.09375 disk.rbd0.out‚0" memory_percent‚99.955153465271 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 7.34765625# proc.memory.rssMB‚ 3035.44140625 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0* disk./.usagePercent‚18.235944551471523! disk.nvme0c0n1.in‚ 162.1796875 network.recv‚795655& disk.nvme0n1.out‚13671.35498046875 disk.sda2.in‚0( disk.nvme0c0n1.out‚13671.35498046875ÁÀ L°‚(:’ ƒåÔ“õ×´ gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚286.239' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.24779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BšTdÔ±‚(:à ƒåÔïùµ·* disk./.usagePercent‚18.236660860552327 disk.drbd1716.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 162.1796875" memory_percent‚99.955153465271 disk.dm-0.in‚ 162.1796875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 cpu‚25.18647247460309 proc.cpu.threads‚17 network.recv‚802106 disk.drbd1716.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚13740.47802734375# disk.dm-0.out‚13740.47802734375çü®'z disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 4.325308e+06$ disk./.usageGB‚271.7394256591797 disk.sda3.in‚0 disk.sda3.out‚ 79.796875 disk.drbd2080.out‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚13740.47802734375# proc.memory.rssMB‚ 3035.44140625 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 79.796875& proc.memory.availableMB‚ 7.34765625 disk.sda.out‚ 79.796875 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.526864051818848o`¤“²‚(:ˆ ’åԔŷµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.626' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆõìJ³‚(:¿ ’åÔÑËá¶& proc.memory.availableMB‚ 7.55078125 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875# disk.dm-0.out‚13787.41162109375 disk.sda.out‚80.1875 network.recv‚806306$ disk./.usageGB‚271.7487678527832 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0$ memory_percent‚99.95391368865967) disk./.usagePercent‚18.23728782300368 disk.drbd2265.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚18.526864051818848 cpu‚25.185458225653452 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.out‚80.1875# proc.memory.rssMB‚ 3035.44140625& disk.nvme0n1.out‚13787.41162109375 disk.dm-1.in‚0 disk.rbd1.out‚ 1.2265625( disk.nvme0c0n1.out‚13787.41162109375 disk.sda.in‚0 network.sent‚ 4.331207e+06 disk.sda3.out‚80.1875 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0\æ5º2´- _wandbruntime‚2215  _runtime‚2215|N·Fɵà _step‚179 ' lr-AdamW/pg1‚0.00019999379351138513 ' lr-AdamW/pg2‚0.00019999379351138513 trainer/global_step‚2249 % _timestamp‚1.7884302371368177e+09 _runtime‚2215.491864385nÿ¨Î¶È _runtime‚2215.491864385 _step‚179 ' lr-AdamW/pg1‚0.00019999379351138513 ' lr-AdamW/pg2‚0.00019999379351138513 trainer/global_step‚2249 % _timestamp‚1.7884302371368177e+09³(‘¬ð2·- _wandbruntime‚2215  _runtime‚2215¿ãú'¸ôO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4785961998336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4782856605696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306333499286528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41100734 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 trainer/global_step‚2249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301547537288192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39729585 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4782856605696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚301547510896640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306333499286528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39729591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257725345883648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884302371375551e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87834496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48104905 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301547510896640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264377296517632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48104905 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39729585 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301310376274819 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39729591 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23682112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4774798620547 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306330367502336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17418619 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264377209173504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257725374018048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17418061 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6651863289856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 _runtime‚2215.492861737 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301313505171703 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440,€ábÿ'¹ùO MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87811386 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4785961998336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23682115 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301547537288192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296535603082240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48081801 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296535577654272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306330367502336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6651922499584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4785961998336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4782856605696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306333499286528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41100734 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 trainer/global_step‚2249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301547537288192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39729585 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4782856605696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚301547510896640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306333499286528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39729591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257725345883648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884302371375551e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87834496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48104905 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301547510896640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264377296517632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48104905 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39729585 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301310376274819 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39729591 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23682112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4774798620547 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306330367502336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17418619 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264377209173504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257725374018048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17418061 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6651863289856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 _runtime‚2215.492861737 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301313505171703 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4784968919552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87872896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48102891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296665000083456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306467105701888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87849760 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39746869 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59196416 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87872896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301679031376384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301679031376384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257837784882176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87849760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48126021 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41118034 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39746869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17426192 trainer/global_step‚2249 $ _timestamp‚1.788430238022535e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4788074325504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41118560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6654780513792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39746875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23692368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264492537261568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4780010882771 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17425669 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4784968919552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87872896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105405952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48102891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296665000083456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306467105701888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87849760 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39746869 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59196416 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87872896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301679031376384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301679031376384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257837784882176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87849760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48126021 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41118034 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39746869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17426192 trainer/global_step‚2249 $ _timestamp‚1.788430238022535e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4788074325504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41118560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6654780513792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39746875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23692368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264492537261568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4780010882771 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17425669 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308122215306240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4883355335671 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308122215306240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313010702018560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89733438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41992840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308122241697792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49139653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263345889090048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6798079358464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4880251866755 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4891592104960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚270144053695488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307888808478711 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884302841312664e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313010702018560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚270143968448512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40593785 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17798075 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308122241697792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41992292 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚ 2262.48660024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313013833802752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4891592104960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4888486712320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40593791 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49162757 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4888486712320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6798136471040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 õƒÈ¾YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308122215306240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4883355335671 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308122215306240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313010702018560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89733438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41992840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308122241697792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49139653 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263345889090048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6798079358464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4880251866755 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4891592104960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚270144053695488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307888808478711 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884302841312664e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313010702018560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚270143968448512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180¸̺é-2Ì- _wandbruntime‚2263  _runtime‚2263Õkn™'ÍýM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308253709394432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89794948 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89771812 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49183873 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4890599026176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24205015 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49160743 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788430285028912e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40611069 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42010119 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263458328088576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89794948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303134850144256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4890599026176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40611075 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303134824716288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49160743 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313144308420608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17805104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308017185376143 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89771812 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329UøáR'΂N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40611069 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42010119 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263458328088576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89794948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303134850144256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4890599026176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40611075 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303134824716288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49160743 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313144308420608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17805104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55002112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308017185376143 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89771812 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308253709394432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89794948 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89771812 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49183873 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4890599026176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24205015 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49160743 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788430285028912e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319691036534784 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚2349 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41457991 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41457985 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6944665089536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314696946107392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319694168318976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314696919715840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41457985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 _runtime‚2309.549090089 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268966432296448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319691036534784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4997222211584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4994116818944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275911038176256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4997222211584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91678600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42884508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24707412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18177645 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18177096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚275911125520384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314460982888835 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309475277775872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 % _timestamp‚1.7884303311938255e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50220609 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314464111785719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚188 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50197505 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4988808581879 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4994116818944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42885060 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50220609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91655490 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4985705112963 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314696946107392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309475303203840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314696919715840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6944605879808 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24707415 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50197505 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319691036534784 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚2349 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41457991 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41457985 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6944665089536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314696946107392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319694168318976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314696919715840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41457985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 _runtime‚2309.549090089 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268966432296448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319691036534784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4997222211584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4994116818944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275911038176256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4997222211584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91678600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42884508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24707412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18177645 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18177096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚275911125520384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314460982888835 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309475277775872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 % _timestamp‚1.7884303311938255e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50220609 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314464111785719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚188 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50197505 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4988808581879 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4994116818944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42885060 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50220609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91655490 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4985705112963 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314696946107392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309475303203840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314696919715840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6944605879808 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24707415 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110¼ÔX“Þ‚(:ˆ ûåÔªŸžµ gpu.0.gpu‚48.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚216.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜Gß‚(:¼ ûåÔäËø¶! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.out‚14201.58203125 disk.rbd0.out‚0 network.sent‚ 4.576627e+06 disk.dm-1.out‚ 84.0390625 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.sda.out‚ 84.0390625 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 162.1796875 disk.sda3.out‚ 84.0390625 disk.nvme0n1.in‚ 162.1796875 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0# disk.nvme0n1.out‚14201.58203125" proc.memory.availableMB‚7.1875 cpu‚25.17904350730595 proc.cpu.threads‚17 network.recv‚839045* disk./.usagePercent‚18.244358751099398 disk.drbd2266.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚99.95613098144531* proc.memory.percent‚18.526864051818848 disk.rbd1.out‚ 1.2265625% disk.nvme0c0n1.out‚14201.58203125% disk./.usageGB‚271.85412979125977# proc.memory.rssMB‚ 3035.44140625þÝ02à- _wandbruntime‚2310  _runtime‚2310ÿª¨Û$áþM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269078843160576 _runtime‚2310.439832611 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91717000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4990917375187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4999334538752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4999334538752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41475269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚189 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6947601932288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50218595 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269078871294976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42902841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309604674777088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41475269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42902295 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319824642936832 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314595617580243 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50218595 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4996229132800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41475275 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276026387993600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4987813906063 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314592488683151 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41475275 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314828440195584 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884303320842519e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319827774734336 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314828440195584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276026473227264 G:DeviceStatsMonitor.on_train_batch_end/ÆUº)active.all.allocated‚91717000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319827774734336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18184630 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314828413804032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180‘iá 'âƒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50218595 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4996229132800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41475275 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276026387993600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57099264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4987813906063 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3272444928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314592488683151 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41475275 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314828440195584 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884303320842519e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319827774734336 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314828440195584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276026473227264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91717000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319827774734336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18184630 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314828413804032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269078843160576 _runtime‚2310.439832611 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91717000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4990917375187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4999334538752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4999334538752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41475269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚189 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6947601932288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50218595 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269078871294976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42902841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309604674777088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41475269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42902295 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319824642936832 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314595617580243 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984½Šª†z2ã- _wandbruntime‚2310  _runtime‚2310¡ ÁvVäÐ 2loss/SRqqt3/assignment_loss‚1.3912441349691815 0loss/SRqqt4/detection_loss‚2.603890214500756 epoch‚1 _runtime‚2310.440861894 /loss/FRt1/assignment_loss‚2.798439117900112 1loss/FBt4/assignment_loss‚0.35179228496797177 0loss/SRqqt1/detection_loss‚1.581275082060269 1loss/SRqqt2/detection_loss‚2.7859520256519317 %loss/total_loss‚52.90042172216059 % _timestamp‚1.7884303320858784e+09 /loss/FRt2/assignment_loss‚5.612647816538811 2loss/SRqqt4/assignment_loss‚2.4747453044200767 1loss/FBt1/assignment_loss‚0.13184341778066647 0loss/FBt2/assignment_loss‚0.3239016420860604 /loss/FBt3/detection_loss‚1.8309923032155404 0loss/FRt3/assignment_loss‚2.7610548973083495 /loss/FRt4/assignment_loss‚5.179115388128492 .loss/FRt4/detection_loss‚3.674744168917338 trainer/global_step‚2349 2loss/SRqqt1/assignment_loss‚1.8056293571633952 /loss/FBt1/detection_loss‚0.9915761686861515 /loss/FBt2/detection_loss‚2.8936788981611077 _step‚190 .loss/FRt2/detection_loss‚2.921901541762054 0loss/FBt3/assignment_loss‚0.3921669688648902 1loss/SRqqt3/detection_loss‚1.3187747249248052 0loss/SRqqt2/assignment_loss‚2.72293550123771 .loss/FRt1/detection_loss‚2.148853317425962 /loss/FRt3/detection_loss‚1.8319337604301316 .loss/FBt4/detection_loss‚2.371333685058814AÈî{[åÕ trainer/global_step‚2349 2loss/SRqqt1/assignment_loss‚1.8056293571633952 /loss/FBt1/detection_loss‚0.9915761686861515 /loss/FBt2/detection_loss‚2.8936788981611077 _step‚190 .loss/FRt2/detection_loss‚2.921901541762054 0loss/FBt3/assignment_loss‚0.3921669688648902 1loss/SRqqt3/detection_loss‚1.3187747249248052 0loss/SRqqt2/assignment_loss‚2.72293550123771 .loss/FRt1/detection_loss‚2.148853317425962 /loss/FRt3/detection_loss‚1.8319337604301316 .loss/FBt4/detection_loss‚2.371333685058814 2loss/SRqqt3/assignment_loss‚1.3912441349691815 0loss/SRqqt4/detection_loss‚2.603890214500756 epoch‚1 _runtime‚2310.440861894 /loss/FRt1/assignment_loss‚2.798439117900112 1loss/FBt4/assignment_loss‚0.35179228496797177 0loss/SRqqt1/detection_loss‚1.581275082060269 1loss/SRqqt2/detection_loss‚2.7859520256519317 %loss/total_loss‚52.90042172216059 % _timestamp‚1.7884303320858784e+09 /loss/FRt2/assignment_loss‚5.612647816538811 2loss/SRqqt4/assignment_loss‚2.4747453044200767 1loss/FBt1/assignment_loss‚0.13184341778066647 0loss/FBt2/assignment_loss‚0.3239016420860604 /loss/FBt3/detection_loss‚1.8309923032155404 0loss/FRt3/assignment_loss‚2.7610548973083495 /loss/FRt4/assignment_loss‚5.179115388128492 .loss/FRt4/detection_loss‚3.674744168917338¾–z ²“æ‚(:ˆ ŠåÔ…Š’µ gpu.0.gpu‚82.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.484' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D£š*Qç‚(:Æ ŠåÔƒ—ö¶ disk.sda1.out‚0# proc.memory.rssMB‚ 3035.44140625 disk.rbd1.out‚ 1.2265625% disk.nvme0n1.out‚14236.7490234375 disk.sda2.in‚0 network.sent‚ 4.673597e+06 disk.drbd2265.out‚0) disk./.usagePercent‚18.24193307971247 disk.drbd2266.in‚0 disk.dm-0.in‚ 162.1796875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0& proc.memory.availableMB‚ 7.14453125 proc.cpu.threads‚17% disk./.usageGB‚271.81798553466797 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚14236.7490234375 disk.sda3.in‚0 disk.sda.out‚ 84.4140625 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 84.4140625$ memory_percent‚99.95639324188232 cpu‚25.178220468831178 disk.drbd2265.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2080.out‚0* proc.memory.percent‚18.526864051818848 network.recv‚844666 disk.sda.in‚0" disk.dm-0.out‚14236.7490234375 disk.sda3.out‚ 84.41406256«™Yžè‚(:“ ™åÔ—€À¶ gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚129.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.974200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙnfUé‚(:Ê ™åÔûÂÖ¶ disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 85.05078125# proc.memory.rssMB‚ 3035.44140625( disk.nvme0c0n1.out‚14300.80126953125 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 4.77184e+06 disk.sda1.in‚0 disk.dm-1.out‚ 85.05078125 disk.sda3.out‚ 85.05078125$ memory_percent‚99.95639324188232* disk./.usagePercent‚18.242848050639846$ disk./.usageGB‚271.8316192626953# disk.dm-0.out‚14300.80126953125 disk.drbd1716.in‚0 disk.sda.in‚0 cpu‚25.17730749959223 disk.drbd2266.out‚0 disk.dm-0.in‚ 162.1796875 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.1796875& disk.nvme0n1.out‚14300.80126953125* proc.memory.percent‚18.526864051818848 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 162.1796875 disk.sda3.in‚0& proc.memory.availableMB‚ 7.14453125 disk.drbd2265.out‚0 network.recv‚851020 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd2080.out‚0~i^ “ê‚(:ˆ ¨åÔ¡€‰µ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚273.707' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žr©µTë‚(:É ¨åÔÖï´¶ disk.rbd1.out‚ 1.2265625* proc.memory.percent‚18.526864051818848( disk.nvme0c0n1.out‚14347.66455078125 disk.sda3.out‚ 85.38671875 disk.drbd2265.in‚0 network.recv‚854385% disk./.usageGB‚271.83482360839844# disk.dm-0.out‚14347.66455078125 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚18.24306309696861 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0$ memory_percent‚99.95489120483398# proc.memory.rssMB‚ 3035.44140625 cpu‚25.17651110110539 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 85.38671875& disk.nvme0n1.out‚14347.66455078125 disk.dm-1.out‚ 85.38671875 disk.drbd2266.in‚0$ proc.memory.availableMB‚7.390625 proc.cpu.threads‚17 disk.drbd2265.out‚0 network.sent‚ 4.776974e+06! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.17968750ÿÞÚ2ì- _wandbruntime‚2356  _runtime‚2356g=ĹÉíà % _timestamp‚1.7884303779905074e+09 _runtime‚2356.345578119 _step‚191 ' lr-AdamW/pg1‚0.00019859233697118716 ' lr-AdamW/pg2‚0.00019859233697118716 trainer/global_step‚2399=ªøÎîÈ trainer/global_step‚2399 % _timestamp‚1.7884303779905074e+09 _runtime‚2356.345578119 _step‚191 ' lr-AdamW/pg1‚0.00019859233697118716 ' lr-AdamW/pg2‚0.00019859233697118716¿DˆG2ï- _wandbruntime‚2356  _runtime‚2356q@Wžù'ðóO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42322191 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51255357 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5102852318208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315945127836672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5102852318208 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42322185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321036286195843 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 $ _timestamp‚1.788430377991241e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315945153264640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5094261828087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43776478 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321271624125440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326374502835200 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚326371371051008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281677751387136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326371371051008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274587003637248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7090688540160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93600652 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43777049 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5091158359171 trainer/global_step‚2399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51255357 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321039415092727 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5099746925568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281677664043008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93577542 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321271624125440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18556416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51278461 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321271650516992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571Ĺ¾ÂñøO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274586975502848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42322191 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18556984 _runtime‚2356.346521379 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896884 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 400486400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25220062 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321271650516992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87344128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326371371051008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27962 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274587003637248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7090688540160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93600652 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43777049 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5091158359171 trainer/global_step‚2399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51255357 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321039415092727 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479241 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5099746925568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281677664043008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93577542 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105392640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321271624125440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18556416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209728 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468916 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51278461 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321271650516992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42322191 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27962 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51255357 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5102852318208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315945127836672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5102852318208 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3272444928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42322185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321036286195843 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 $ _timestamp‚1.788430377991241e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105392640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315945153264640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5094261828087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43776478 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321271624125440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326374502835200 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚326371371051008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281677751387136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43794157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5096370621395 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51299577 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5101859239424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639052 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321403118213632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326508109250560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316074524837888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51276447 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42339475 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639052 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93615916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5104964645376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25230315 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274699386366976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚2357.236577664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321170920887251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321403144605184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274699414501376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281792989742080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18564379 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316074550265856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚193 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326504977453056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326508109250560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5104964645376 % _timestamp‚1.7884303788806438e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18563842 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5101859239424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93615916 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42339469 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42339469 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51299577 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43794697 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5093267152271 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321403144605184 ¶° 'ôƒN C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93615916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5104964645376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25230315 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274699386366976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚2357.236577664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321170920887251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2399 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131797504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321403144605184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274699414501376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281792989742080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18564379 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316074550265856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚193 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326504977453056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326508109250560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5104964645376 % _timestamp‚1.7884303788806438e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18563842 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5101859239424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93615916 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42339469 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469124 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42339469 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51299577 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43794697 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5093267152271 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321403144605184 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131797504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43794157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5096370621395 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51299577 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897092 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5101859239424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639052 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321403118213632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 400486400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105405952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23130 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326508109250560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316074524837888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51276447 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27962 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479241 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42339475 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639052 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878Áj¾~2õ- _wandbruntime‚2357  _runtime‚2357QòÈi[öÕ /loss/FBt2/detection_loss‚3.9593970605305264 %loss/total_loss‚53.93317446461206 /loss/FRt4/assignment_loss‚5.263559379075703 2loss/SRqqt1/assignment_loss‚1.6557573274266524 /loss/FRt3/detection_loss‚2.1777999694360064 2loss/SRqqt4/assignment_loss‚1.9915332504148995 1loss/FBt2/assignment_loss‚0.38952428521588445 1loss/FBt4/assignment_loss‚0.18768837805833377 /loss/FRt4/detection_loss‚3.4344771127951774 .loss/FBt3/detection_loss‚4.702550798654556 1loss/FBt1/assignment_loss‚0.21501322268764356 /loss/FRt1/detection_loss‚2.0258584017200127 1loss/SRqqt3/detection_loss‚1.6103969603580863 0loss/SRqqt4/detection_loss‚2.803568621831281 epoch‚1 _runtime‚2357.238002493 trainer/global_step‚2399 2loss/SRqqt2/assignment_loss‚1.8320399191110366 1loss/SRqqt3/assignment_loss‚1.273312497736984 1loss/FBt3/assignment_loss‚0.13323117463854336 0loss/SRqqt2/detection_loss‚2.786352530602486 /loss/FBt1/detection_loss‚1.0365172666243532 _step‚194 /loss/FRt2/detection_loss‚2.6845645495823454 /loss/FRt1/assignment_loss‚2.496223602976118 1loss/SRqqt1/detection_loss‚1.2731062088344918 .loss/FBt4/detection_loss‚2.600958636403084 % _timestamp‚1.7884303788830287e+09 /loss/FRt2/assignment_loss‚5.094729334967477 0loss/FRt3/assignment_loss‚2.3050139749303775R9ø`÷Ú 0loss/SRqqt2/detection_loss‚2.786352530602486 /loss/FBt1/detection_loss‚1.0365172666243532 _step‚194 /loss/FRt2/detection_loss‚2.6845645495823454 /loss/FRt1/assignment_loss‚2.496223602976118 1loss/SRqqt1/detection_loss‚1.2731062088344918 .loss/FBt4/detection_loss‚2.600958636403084 % _timestamp‚1.7884303788830287e+09 /loss/FRt2/assignment_loss‚5.094729334967477 0loss/FRt3/assignment_loss‚2.3050139749303775 /loss/FBt2/detection_loss‚3.9593970605305264 %loss/total_loss‚53.93317446461206 /loss/FRt4/assignment_loss‚5.263559379075703 2loss/SRqqt1/assignment_loss‚1.6557573274266524 /loss/FRt3/detection_loss‚2.1777999694360064 2loss/SRqqt4/assignment_loss‚1.9915332504148995 1loss/FBt2/assignment_loss‚0.38952428521588445 1loss/FBt4/assignment_loss‚0.18768837805833377 /loss/FRt4/detection_loss‚3.4344771127951774 .loss/FBt3/detection_loss‚4.702550798654556 1loss/FBt1/assignment_loss‚0.21501322268764356 /loss/FRt1/detection_loss‚2.0258584017200127 1loss/SRqqt3/detection_loss‚1.6103969603580863 0loss/SRqqt4/detection_loss‚2.803568621831281 epoch‚1 _runtime‚2357.238002493 trainer/global_step‚2399 2loss/SRqqt2/assignment_loss‚1.8320399191110366 1loss/SRqqt3/assignment_loss‚1.273312497736984 1loss/FBt3/assignment_loss‚0.13323117463854336Âwwq5žø‚(:“ ·åԮ߳µ gpu.0.gpu‚57.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚293.084' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.616800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G pIù‚(:¾ ·åÔ‚¦í· disk.dm-1.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3035.4453125& disk.nvme0n1.out‚14430.42919921875 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda3.out‚86.0625 network.sent‚ 4.78302e+06 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0$ memory_percent‚99.72624778747559 disk.nvme0n1.in‚ 162.1796875 disk.rbd1.out‚ 1.2265625! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.sda2.in‚0 network.recv‚858952% disk./.usageGB‚271.84787368774414 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0# disk.dm-0.out‚14430.42919921875 disk.drbd1716.in‚0 disk.drbd2080.in‚0& proc.memory.availableMB‚ 44.8515625 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.dm-1.out‚86.0625* disk./.usagePercent‚18.243938898743245 disk.drbd2265.out‚0 disk.sda.out‚86.0625 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚14430.42919921875* proc.memory.percent‚18.526887893676758 cpu‚25.17558330253021 ô/é›ú‚(: ÆåÔ¹œ·¶ gpu.0.gpu‚98.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚297.7' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.53999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Ä3îSû‚(:È ÆåÔ¼óк network.sent‚ 4.891532e+06* disk./.usagePercent‚18.239869146971426 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0& proc.memory.availableMB‚ 44.5546875 disk.sda3.out‚ 86.47265625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 1.2265625! disk.nvme0c0n1.in‚ 162.1796875" disk.dm-0.out‚14476.9970703125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.out‚ 86.47265625 network.recv‚864795 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.1796875 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3035.4453125$ disk./.usageGB‚271.7872314453125 disk.rbd0.out‚0 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚14476.9970703125$ memory_percent‚99.72805976867676* proc.memory.percent‚18.526887893676758 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.drbd2265.in‚0% disk.nvme0n1.out‚14476.9970703125 disk.sda.out‚ 86.47265625 cpu‚25.194651638759932‰U»ü‚(:„ ÕåÔóÀá´ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ª´ Ùý‚(: ÕåÔ°˜å¸ disk.sda2.in‚0 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚14545.7392578125 disk.drbd1716.out‚0 cpu‚25.219253585348635 disk.rbd0.in‚0" disk.dm-0.out‚14545.73925781ƒé –t25 disk.drbd1716.in‚0 disk.drbd2080.out‚0) disk./.usagePercent‚18.24532313148088 disk.dm-1.in‚0 disk.sda3.out‚ 87.0234375 network.recv‚869306$ disk./.usageGB‚271.8684997558594 disk.sda.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0$ memory_percent‚98.64516258239746' proc.memory.availableMB‚ 221.9765625 proc.memory.rssMB‚ 2963.21875 disk.sda.out‚ 87.0234375 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.sent‚ 4.897689e+06% disk.nvme0n1.out‚14545.7392578125 disk.rbd1.out‚ 1.2265625! disk.nvme0c0n1.in‚ 162.1796875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 87.0234375 disk.dm-0.in‚ 162.1796875 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0) proc.memory.percent‚18.08605194091797®Y«Ãžþ‚(:“ äåÔÅÝζ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚305.179' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.035799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß³»JQÿ‚(:Æ äåÔ“‚ú¶$ disk./.usageGB‚271.8791961669922 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.rbd1.out‚ 1.2265625 disk.dm-0.in‚ 162.1796875 disk.sda1.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚14591.2158203125" proc.memory.rssMB‚ 3035.4453125 network.recv‚872958" disk.dm-0.out‚14591.2158203125 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 87.46484375 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚ 87.46484375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.1796875* proc.memory.percent‚18.526887893676758 cpu‚25.24570357558172 network.sent‚ 4.903085e+06 disk.drbd2080.out‚0 disk.dm-1.out‚ 87.46484375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚14591.2158203125 disk.drbd2265.in‚0$ memory_percent‚99.08733367919922% proc.memory.availableMB‚ 149.53125 proc.cpu.threads‚17* disk./.usagePercent‚18.246040976606892 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0ËSiK2€- _wandbruntime‚2428  _runtime‚2428ès‰4c*ÝT KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚327722549163159 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚43297284 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353900032 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚25888406 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚322566226700288 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44823040 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5167406330368 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27962 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131225088 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚328160051778048 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚95335055 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚328160051778048 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚328160076072448 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18276 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚95353336 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5165076724736 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚52037771 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479241 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚2417 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚280411369592832 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113576448 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚2428.757662946 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚18934634 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 25086444032 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7171092552704 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚95353336 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚121 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚327724900184807 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18276 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚52037771 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚18934755 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690832 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878631424 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5167406330368 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚322566203369472 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18281 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚333325128502784 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329605632 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚287582444496896 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021648 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18281 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28215 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚287582575721984 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚44823163 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚968 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚328160076072448 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚95335055 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚43297289 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3275873280 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚333327482402816 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚280411351944192 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚123 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚52056047 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 _step‚195h L¸h*‚âT UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚52037771 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479241 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚2417 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚280411369592832 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113576448 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚2428.757662946 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚18934634 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 25086444032 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7171092552704 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚95353336 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚121 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚327724900184807 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18276 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚52037771 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚18934755 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690832 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878631424 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5167406330368 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚322566203369472 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18281 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚333325128502784 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329605632 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚287582444496896 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021648 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18281 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28215 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚287582575721984 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878631424 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚44823163 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚968 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚328160076072448 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚95335055 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚43297289 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3275873280 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚333327482402816 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚280411351944192 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚123 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚52056047 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 _step‚195 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚327722549163159 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚43297284 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353900032 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚25888406 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚322566226700288 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44823040 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5167406330368 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27962 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131225088 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚328160051778048 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚95335055 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚328160051778048 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚328160076072448 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18276 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚95353336 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5165076724736Ão/2ƒ- _wandbruntime‚2429  _runtime‚2429ß6u6!„æR YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚333417573029888 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚328249668682240 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329605632 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚328249668682240 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27962 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693088 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3275873280 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚333415219129856 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 113576448 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚44837672 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18281 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚95355947 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚52063741 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5159169936167 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878631424 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚18939600 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚280486628669440 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28215 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚122 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5156842245335 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28215 NHDeeÃ}k6viceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚25898072 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44837548 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚328249644387840 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚43310487 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚287658219239424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7171704146432 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131225088 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚43310482 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353900032 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚95374228 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5167904347648 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329605632O[UŒq)…ëR TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚44837672 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18281 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚95355947 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚52063741 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5159169936167 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878631424 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚18939600 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚280486628669440 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28215 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚122 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5156842245335 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28215 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚25898072 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44837548 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚328249644387840 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚43310487 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚287658219239424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7171704146432 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131225088 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚43310482 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353900032 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚95374228 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5167904347648 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329605632 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878631424 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚333417573029888 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚328249668682240 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329605632 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚328249668682240 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27962 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693088 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3275873280 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚333415219129856 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 113576448Ä•ê|Êœ†‚(:‘ óåÔÛ±¿¶ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.713' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.942600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žZ=S‡‚(:È óåÔ÷ÿò¶ disk.drbd2080.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0" proc.memory.rssMB‚ 3035.4453125 cpu‚25.269552737042375 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 88.0703125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 162.1796875" disk.dm-0.out‚14645.5966796875! disk.nvme0c0n1.in‚ 162.1796875$ memory_percent‚99.08959865570068( proc.memory.availableMB‚ 149.16015625 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.2265625 network.sent‚ 4.908996e+06 disk.nvme0n1.in‚ 162.1796875' disk.nvme0c0n1.out‚14645.5966796875% disk.nvme0n1.out‚14645.5966796875 disk.dm-1.in‚0 disk.drbd2266.in‚0* disk./.usagePercent‚18.247833797369175% disk./.usageGB‚271.90591049194336 disk.sda3.out‚ 88.0703125 disk.sda1.in‚0* proc.memory.percent‚18.526887893676758 network.recv‚877337 disk.drbd2265.in‚0 disk.dm-1.out‚ 88.0703125Uíi2ˆ- _wandbruntime‚2444  _runtime‚2444ûòQH‰¥¦ @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5610119104385376 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3606199324131012 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07524310052394867 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3609161376953125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09332640469074249 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 jet/accuracy_5_of_6‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22566543519496918 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20804595947265625 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1186719760298729 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.33302608132362366 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.59375 jet/accuracy_7_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5713369846343994 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09443541616201401 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9100000262260437 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21972204744815826 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 particle/accuracy_2_of_6‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9544841051101685 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3921073377132416 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3968600928783417 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.41273143887519836 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,jet/accuracy_4_of_4‚0.003212909447029233 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_2_of_2‚0.09193450957536697 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3408859968185425 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24635028839111328 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9758376479148865 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08640109747648239 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28995954990386963 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3982206881046295 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3469782769680023 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4197257459163666 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.30426356196403503 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2881944477558136 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06710108369588852 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5329235792160034 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 _runtime‚2444.349045185 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9465085864067078 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.98628169298172 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1473582535982132 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4382183849811554 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3225477933883667 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47816088795661926 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9800488948822021 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5673235058784485 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5358378291130066 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5673611164093018 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6333305835723877 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28136295080184937 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24722224473953247 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9866074919700623 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11841144412755966 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9851504564285278 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.026268817484378815 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17382179200649261 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9704378247261047 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13419176638126373 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3066040277481079 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.053789667785167694 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9815732836723328 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.11651235073804855 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9805065989494324 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008484505116939545 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14954259991645813 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4985385239124298 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.42292696237564087 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12309393286705017 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9809272289276123 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.27272728085517883 +jet/accuracy_2_of_3‚0.19958725571632385 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 epoch‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997893750667572 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01077906135469675 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003379286266863346 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2305959165096283 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9769214391708374 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.32894736528396606 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12345679849386215 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.24804022908210754 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9793686270713806 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9501736164093018 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9568261504173279 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9724485874176025 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47030192613601685 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007245120592415333 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.47952863574028015 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9859703779220581 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3733922839164734 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10584258288145065 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6693170666694641 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9389880895614624 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.40290823578834534 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39524129033088684 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8666666746139526 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.981017529964447 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4340820610523224 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21153892576694489 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 _step‚197 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020892035216093063 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04885632172226906 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007186026778072119 *jet/accuracy_2_of_4‚0.3210441470146179 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 +jet/accuracy_3_of_5‚0.07438556104898453 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5282421112060547 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 +jet/accuracy_3_of_6‚0.09615384787321091 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3881225883960724 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3501857817173004 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 particle/accuracy_4_of_6‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08821791410446167 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26611098647117615 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23496462404727936 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 +ðˆæšK Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4179117977619171 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.32499998807907104 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17976367473602295 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4050014615058899 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5601718425750732 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 %particle/f_score‚0.30570188164711 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02323230914771557 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5090576410293579 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5272880792617798 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47562217712402344 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962868690490723 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15538935363292694 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10824371874332428 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9591953754425049 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13434816896915436 *jet/accuracy_1_of_5‚0.8735024333000183 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12561091780662537 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.009408602491021156 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.42018479108810425 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15952037274837494 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 *jet/accuracy_1_of_2‚0.7049046754837036 (particle/accuracy‚0.8332763910293579 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13997048139572144 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4271780252456665 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14627492427825928 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.47330978512763977 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36721721291542053 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01708909310400486 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.14814813435077667 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3579792082309723 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4243827164173126 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4510968327522278 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9922487735748291 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12461666762828827 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.12398988753557205 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.38137611746788025 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9430931806564331 jet/accuracy_3_of_7‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5911803245544434 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18182432651519775 +jet/accuracy_3_of_3‚0.01981356553733349 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 % _timestamp‚1.7884304659918196e+09 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.45771297812461853 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9710801243782043 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971429109573364 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1344827562570572 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19338247179985046 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14978870749473572 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47663938999176025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5515873432159424 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4852980077266693 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ) Purity/*FRt4SRqqt0FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.102229043841362 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4510968327522278 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04004940018057823 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0017636684933677316 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.992137610912323 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3553376793861389 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9910849332809448 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9881170392036438 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4448067247867584 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09978415071964264 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13003277778625488 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10743135213851929 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.26923078298568726 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.060606058686971664 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9881365895271301 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.350326806306839 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46000000834465027 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760234355926514 particle/accuracy_2_of_3‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.39844170212745667 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9089835286140442 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8084666728973389 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.38810473680496216 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4259258806705475 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚0.5 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24832043051719666 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 )loss/val_total_loss‚51.38368376058643 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09482911974191666 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121212482452393 +particle/specificity‚0.9438873529434204 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 particle/accuracy_3_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997893750667572 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10173749923706055 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17087768018245697 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12572155892848969 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0980832502245903 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4469708800315857 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 trainer/global_step‚2417 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5600484013557434 ) Purity/*FRt4SRqqt*FBt/FRt_purity‚0.25 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14164021611213684 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0038803413044661283 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5491218566894531 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_4‚0.037378761917352676 ,particle/sensitivity‚0.23538683354854584 jet/accuracy_6_of_6‚0 jet/accuracy_5_of_7‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4951707124710083 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8070590496063232 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39233431220054626 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5794354677200317 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4088617265224457 jet/accuracy_5_of_5‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06105957180261612 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9674036502838135 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4535507559776306 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10771797597408295 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.28177085518836975 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1409909725189209 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2945767045021057 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007615840528160334 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3396100103855133 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9915463924407959 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.25386902689933777 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2196969836950302 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08821791410446167 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22297300398349762 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12246432900428772 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2291666567325592 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.521889865398407 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1111111044883728 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8725823163986206 particle/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13793104887008667 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4431798458099365 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.44479066133499146 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5û‚NÔ4Šª¦ =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020892035216093063 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04885632172226906 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007186026778072119 *jet/accuracy_2_of_4‚0.3210441470146179 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 +jet/accuracy_3_of_5‚0.07438556104898453 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5282421112060547 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 +jet/accuracy_3_of_6‚0.09615384787321091 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3881225883960724 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3501857817173004 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 particle/accuracy_4_of_6‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08821791410446167 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26611098647117615 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23496462404727936 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4179117977619171 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.32499998807907104 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17976367473602295 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4050014615058899 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5601718425750732 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 %particle/f_score‚0.30570188164711 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02323230914771557 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5090576410293579 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5272880792617798 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47562217712402344 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9962868690490723 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15538935363292694 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10824371874332428 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9591953754425049 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13434816896915436 *jet/accuracy_1_of_5‚0.8735024333000183 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12561091780662537 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.009408602491021156 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.42018479108810425 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15952037274837494 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 *jet/accuracy_1_of_2‚0.7049046754837036 (particle/accuracy‚0.8332763910293579 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13997048139572144 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4271780252456665 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14627492427825928 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.47330978512763977 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36721721291542053 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01708909310400486 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.14814813435077667 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3579792082309723 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4243827164173126 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4510968327522278 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9922487735748291 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12461666762828827 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.12398988753557205 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.38137611746788025 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9430931806564331 jet/accuracy_3_of_7‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5911803245544434 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18182432651519775 +jet/accuracy_3_of_3‚0.01981356553733349 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 % _timestamp‚1.7884304659918196e+09 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.45771297812461853 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9710801243782043 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971429109573364 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1344827562570572 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19338247179985046 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14978870749473572 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47663938999176025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5515873432159424 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4852980077266693 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ) Purity/*FRt4SRqqt0FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.102229043841362 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4510968327522278 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04004940018057823 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0017636684933677316 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.992137610912323 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3553376793861389 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9910849332809448 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9881170392036438 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4448067247867584 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09978415071964264 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13003277778625488 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10743135213851929 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.26923078298568726 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.060606058686971664 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9881365895271301 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8"Purity/0FRt3SRqqt0FBt/SRqqt_purityýõrR]^‚0.350326806306839 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46000000834465027 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760234355926514 particle/accuracy_2_of_3‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.39844170212745667 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9089835286140442 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8084666728973389 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.38810473680496216 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4259258806705475 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚0.5 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24832043051719666 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 )loss/val_total_loss‚51.38368376058643 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09482911974191666 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121212482452393 +particle/specificity‚0.9438873529434204 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 particle/accuracy_3_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997893750667572 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10173749923706055 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17087768018245697 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12572155892848969 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0980832502245903 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4469708800315857 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 trainer/global_step‚2417 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5600484013557434 ) Purity/*FRt4SRqqt*FBt/FRt_purity‚0.25 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14164021611213684 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0038803413044661283 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5491218566894531 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_4‚0.037378761917352676 ,particle/sensitivity‚0.23538683354854584 jet/accuracy_6_of_6‚0 jet/accuracy_5_of_7‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4951707124710083 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8070590496063232 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39233431220054626 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5794354677200317 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4088617265224457 jet/accuracy_5_of_5‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06105957180261612 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9674036502838135 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4535507559776306 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10771797597408295 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.28177085518836975 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1409909725189209 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2945767045021057 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007615840528160334 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3396100103855133 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9915463924407959 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.25386902689933777 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2196969836950302 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08821791410446167 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22297300398349762 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12246432900428772 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2291666567325592 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.521889865398407 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1111111044883728 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8725823163986206 particle/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13793104887008667 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4431798458099365 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.44479066133499146 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5610119104385376 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3606199324131012 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07524310052394867 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3609161376953125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09332640469074249 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 jet/accuracy_5_of_6‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22566543519496918 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20804595947265625 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1186719760298729 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.33302608132362366 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.59375 jet/accuracy_7_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5713369846343994 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09443541616201401 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9100000262260437 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21972204744815826 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 particle/accuracy_2_of_6‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9544841051101685 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3921073377132416 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3968600928783417 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.41273143887519836 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,jet/accuracy_4_of_4‚0.003212909447029233 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_2_of_2‚0.09193450957536697 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3408859968185425 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24635028839111328 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9758376479148865 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08640109747648239 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28995954990386963 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3982206881046295 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3469782769680023 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4197257459163666 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.30426356196403503 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2881944477558136 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06710108369588852 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5329235792160034 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 _runtime‚2444.349045185 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9465085864067078 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.98628169298172 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1473582535982132 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4382183849811554 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3225477933883667 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47816088795661926 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9800488948822021 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5673235058784485 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5358378291130066 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5673611164093018 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6333305835723877 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28136295080184937 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24722224473953247 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9866074919700623 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11841144412755966 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9851504564285278 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.026268817484378815 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17382179200649261 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9704378247261047 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13419176638126373 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3066040277481079 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.053789667785167694 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9815732836723328 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.11651235073804855 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9805065989494324 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008484505116939545 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14954259991645813 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4985385239124298 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.42292696237564087 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12309393286705017 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9809272289276123 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.27272728085517883 +jet/accuracy_2_of_3‚0.19958725571632385 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 epoch‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997893750667572 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01077906135469675 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003379286266863346 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2305959165096283 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9769214391708374 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.32894736528396606 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12345679849386215 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.24804022908210754 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9793686270713806 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9501736164093018 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9568261504173279 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9724485874176025 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47030192613601685 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007245120592415333 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.47952863574028015 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9859703779220581 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3733922839164734 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10584258288145065 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6693170666694641 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9389880895614624 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.40290823578834534 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39524129033088684 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8666666746139526 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.981017529964447 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4340820610523224 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21153892576694489 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 _step‚197Åôl„‘‹‚(:† ‚‘åÔÒÌœµ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.432' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ó¬FŒ‚(:» ‚‘åÔÒÚØ·% disk.nvme0n1.out‚14695.7138671875 disk.sda1.in‚0 disk.sda.out‚88.6875 disk.rbd0.out‚0& proc.memory.availableMB‚ 40.3984375 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.drbd1716.out‚0 cpu‚25.29329672741512 disk.drbd2266.in‚0 disk.drbd2265.out‚0$ memory_percent‚99.75342750549316# proc.memory.rssMB‚ 3145.51953125) proc.memory.percent‚19.19872760772705 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.2265625 network.sent‚ 5.017081e+06$ disk./.usageGB‚271.9079780578613 disk.sda2.out‚0 disk.dm-1.out‚88.6875' disk.nvme0c0n1.out‚14695.7138671875 disk.sda1.out‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2080.out‚0* disk./.usagePercent‚18.247972553452733 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 162.1796875" disk.dm-0.out‚14695.7138671875 disk.sda3.in‚0 disk.sda3.out‚88.6875 disk.sda.in‚0 network.recv‚883558§T‚(:’ ‘‘åÔ‹Û‹µ gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.243' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.04859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÝERYŽ‚(:Î ‘‘åÔنѷ disk.dm-1.out‚ 89.30078125 disk.drbd2266.in‚0 network.sent‚ 5.141535e+06 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.285393018144116 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚21004.64794921875 disk.sda2.out‚0) proc.memory.availableMB‚ 4473.31640625 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd1716.out‚0# disk.dm-0.out‚21004.64794921875 disk.sda3.out‚ 89.30078125 disk.sda1.out‚0 disk.nvme0n1.in‚ 162.1796875( disk.nvme0c0n1.out‚21004.64794921875# proc.memory.rssMB‚ 3041.44921875 network.recv‚890494# disk./.usageGB‚274.131290435791 disk.drbd2080.in‚0 disk.sda.out‚ 89.30078125 disk.dm-1.in‚0 disk.rbd1.in‚0* proc.memory.percent‚18.563532829284668 disk.drbd2080.out‚0* disk./.usagePercent‚18.397180912618936 disk.rbd0.out‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.in‚ 162.1796875 disk.rbd1.out‚ 1.2265625$ memory_percent‚72.69704341888428~¡“‚(:ˆ  ‘åÔþÓø´ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚277.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûa÷YW‚(:Ì  ‘åÔê’é· network.sent‚ 5.146862e+06$ disk./.usageGB‚274.1546745300293 disk.dm-1.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 4474.2890625 cpu‚25.28397713455349 disk.rbd0.out‚0 network.recv‚893952* disk./.usagePercent‚18.398750238803828! disk.nvme0c0n1.in‚ 162.1796875 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0$ memory_percent‚72.69110679626465( disk.nvme0c0n1.out‚21055.13623046875 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚21055.13623046875 disk.dm-1.out‚ 89.60546875 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.2265625 disk.dm-0.in‚ 162.1796875 disk.sda.in‚0 disk.sda.out‚ 89.60546875 disk.sda2.in‚0# proc.memory.rssMB‚ 3040.46484375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.sda3.out‚ 89.60546875# disk.dm-0.out‚21055.13623046875 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚18.55752468109131 disk.drbd2265.in‚0”-w‹2‘- _wandbruntime‚2487  _runtime‚2487c[~É’Ã ' lr-AdamW/pg1‚0.00019808491025279147 ' lr-AdamW/pg2‚0.00019808491025279147 trainer/global_step‚2449 % _timestamp‚1.7884305089237194e+09 _runtime‚2487.278830588 _step‚198¤ÝíÎ“È % _timestamp‚1.7884305089237194e+09 _runtime‚2487.278830588 _step‚198 ' lr-AdamW/pg1‚0.00019808491025279147 ' lr-AdamW/pg2‚0.00019808491025279147 trainer/global_step‚2449Æ´ÜT2”- _wandbruntime‚2487  _runtime‚2487Ú!d] •óO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338730559743488 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96817722 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 trainer/global_step‚2449 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45582645 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7270666889728 $ _timestamp‚1.788430508924543e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333490658849792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52799863 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44017865 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚495 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327766802713600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52799863 E;DeviceStatsMonit§½âù\or.on_train_batch_start/allocation.all.peak‚28216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19241600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚2487.279849661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5236769108992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26341537 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5236769108992 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7270618166272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292220586124800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284949996092928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5227905166573 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338730559743488 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52822968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333490685241344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45583140 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44017865 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44017859 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338727427958784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96840833 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26341540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚492 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284949967958528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332994707880173 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52799863 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44017865 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚495 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327766802713600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52799863 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19241600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚2487.279849661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5236769108992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26341537 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5236769108992 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7270618166272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292220586124800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284949996092928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5227905166573 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338730559743488 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52822968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333490685241344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45583140 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44017865 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44017859 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338727427958784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96840833 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26341540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚492 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284949967958528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332994707880173 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45582645 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7270666889728 $ _timestamp‚1.788430508924543e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180ÇÌ®eò2—- _wandbruntime‚2488  _runtime‚2488Í~Î'˜þM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5238881422848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238881422848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96856096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26351793 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5233117428801 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52844084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52820953 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333622179329536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338861034360832 trainer/global_step‚2449 _runtime‚2488.420797163 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44035143 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96879233 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45601118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285062406957056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52844084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333622152937984 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76844544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333126213674489 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5230013959673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44035149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285062378822656 % _timestamp‚1.7884305100649614e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96856096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327896225142784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26351790 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333129342571585 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45600636 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7273634794496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333622152937984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5241986829312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48710144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0/NM™ƒN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52844084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333622152937984 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76844544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333126213674489 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5230013959673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44035149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285062378822656 % _timestamp‚1.7884305100649614e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96856096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327896225142784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26351790 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333129342571585 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45600636 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7273634794496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333622152937984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5241986829312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48710144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5238881422848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238881422848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96856096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26351793 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5233117428801 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52844084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52820953 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333622179329536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338861034360832 trainer/global_step‚2449 _runtime‚2488.420797163 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44035143 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96879233 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45601118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285062406957056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456ÈNZ)Œ2š- _wandbruntime‚2488  _runtime‚2488‚iY=Y›Ó /loss/FRt1/assignment_loss‚2.511303258320642 /loss/FRt2/assignment_loss‚4.715296617774076 1loss/FBt4/assignment_loss‚0.41423373737117425 .loss/FRt2/detection_loss‚2.520423958467883 1loss/SRqqt2/detection_loss‚2.6840174515401163 /loss/FBt3/detection_loss‚2.5476947852543423 1loss/SRqqt2/assignment_loss‚1.901392430547745 /loss/FRt1/detection_loss‚2.0425729675898476 /loss/FBt1/detection_loss‚1.3640949478034268 2loss/SRqqt4/assignment_loss‚1.9226318818650074 0loss/FBt3/assignment_loss‚0.2818136105945866 1loss/SRqqt4/detection_loss‚2.4596331268548965 0loss/FBt1/assignment_loss‚0.2520265503358518 1loss/SRqqt3/detection_loss‚1.2852758078635493 .loss/FRt3/assignment_loss‚2.11216074553403 /loss/FBt2/detection_loss‚3.3498446260179793 /loss/FRt3/detection_loss‚1.9182354014028202 2loss/SRqqt3/assignment_loss‚1.2865903092477518 /loss/FRt4/detection_loss‚2.9318042684484413 %loss/total_loss‚48.83512313784656 trainer/global_step‚2449 % _timestamp‚1.7884305100669668e+09 _runtime‚2488.421962843 _step‚201 /loss/FRt4/assignment_loss‚5.283914857440525 2loss/SRqqt1/assignment_loss‚1.3628759186376225 0loss/FBt2/assignment_loss‚0.5311642757379299 1loss/SRqqt1/detection_loss‚1.4605693218382922 /loss/FBt4/detection_loss‚1.6955522813580253 epoch‚2Ï ±q^œØ 0loss/FBt1/assignment_loss‚0.2520265503358518 1loss/SRqqt3/detection_loss‚1.2852758078635493 .loss/FRt3/assignment_loss‚2.11216074553403 /loss/FBt2/detection_loss‚3.3498446260179793 /loss/FRt3/detection_loss‚1.9182354014028202 2loss/SRqqt3/assignment_loss‚1.2865903092477518 /loss/FRt4/detection_loss‚2.9318042684484413 %loss/total_loss‚48.83512313784656 trainer/global_step‚2449 % _timestamp‚1.7884305100669668e+09 _runtime‚2488.421962843 _step‚201 /loss/FRt4/assignment_loss‚5.283914857440525 2loss/SRqqt1/assignment_loss‚1.3628759186376225 0loss/FBt2/assignment_loss‚0.5311642757379299 1loss/SRqqt1/detection_loss‚1.4605693218382922 /loss/FBt4/detection_loss‚1.6955522813580253 epoch‚2 /loss/FRt1/assignment_loss‚2.511303258320642 /loss/FRt2/assignment_loss‚4.715296617774076 1loss/FBt4/assignment_loss‚0.41423373737117425 .loss/FRt2/detection_loss‚2.520423958467883 1loss/SRqqt2/detection_loss‚2.6840174515401163 /loss/FBt3/detection_loss‚2.5476947852543423 1loss/SRqqt2/assignment_loss‚1.901392430547745 /loss/FRt1/detection_loss‚2.0425729675898476 /loss/FBt1/detection_loss‚1.3640949478034268 2loss/SRqqt4/assignment_loss‚1.9226318818650074 0loss/FBt3/assignment_loss‚0.2818136105945866 1loss/SRqqt4/detection_loss‚2.4596331268548965É÷f(Vž‚(:“ ¯‘åÔç—±¶ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚287.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.493399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’N=Jž‚(:¿ ¯‘åÔïÓÚ¶$ memory_percent‚72.69506454467773 disk.rbd1.out‚1.375 disk.dm-0.in‚ 162.1796875 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 cpu‚25.282505914567732) disk./.usagePercent‚18.39987820799969 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 network.sent‚ 5.152798e+06% disk.nvme0c0n1.out‚21116.82421875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.1796875 disk.sda2.out‚0 disk.dm-1.out‚ 90.73046875 disk.drbd2266.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0) proc.memory.percent‚18.55752468109131 proc.cpu.threads‚17% disk./.usageGB‚274.17148208618164 disk.sda3.out‚ 90.73046875 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 90.73046875# disk.nvme0n1.out‚21116.82421875 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.out‚21116.82421875 disk.drbd2265.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 4473.640625# proc.memory.rssMB‚ 3040.46484375 network.recv‚898081 disk.drbd1716.out‚0n¼«Ÿ‚(:’ ¾‘åÔ»žä¶ gpu.0.gpu‚81.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.732' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.54640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*¢ÉqF ‚(:» ¾‘åÔªíï¶ disk.drbd2080.in‚0$ memory_percent‚72.69210815429688) proc.memory.percent‚18.55752468109131 proc.cpu.threads‚17 network.sent‚ 5.26155e+06 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 91.19140625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚4474.125) disk./.usagePercent‚18.40085436472775 disk.sda2.out‚0% disk.nvme0c0n1.out‚21162.79296875 disk.dm-0.in‚ 162.1796875 cpu‚25.281124429552655 disk.dm-1.out‚ 91.19140625 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 162.1796875 network.recv‚904121 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚21162.79296875! disk.nvme0c0n1.in‚ 162.1796875 disk.dm-0.out‚21162.79296875# proc.memory.rssMB‚ 3040.46484375 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0% disk./.usageGB‚274.18602752685547 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda.in‚0 disk.rbd1.out‚1.375 disk.sda3.out‚ 91.19140625 disk.sda1.in‚0’dž¡‚(:“ Í‘åÔ‡ù†µ gpu.0.gpu‚56.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚128.792' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.758399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w¾¥%G¢‚(:¼ Í‘åÔßñè¶ disk.sda.out‚91.9375 disk.dm-1.in‚0" disk.dm-0.out‚21215.5087890625 disk.sda1.in‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 162.1796875 cpu‚25.27986181910038 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 162.1796875% disk.nvme0n1.out‚21215.5087890625 disk.sda2.out‚0 disk.dm-1.out‚91.9375$ memory_percent‚72.69363403320312 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚21215.5087890625 disk.sda3.in‚0 network.sent‚ 5.267856e+06* disk./.usagePercent‚18.401716342095536 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.4072265625 disk.sda3.out‚91.9375$ proc.memory.availableMB‚4473.875# proc.memory.rssMB‚ 3040.46484375$ disk./.usageGB‚274.1988716125488 disk.drbd1716.in‚0 disk.dm-0.in‚ 162.1796875 network.recv‚908858 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.55752468109131ìWU¡2£- _wandbruntime‚2534  _runtime‚2534 Aÿaɤà % _timestamp‚1.7884305561497593e+09 _runtime‚2534.504783732 _step‚202 ' lr-AdamW/pg1‚0.00019755751902113912 ' lr-AdamW/pg2‚0.00019755751902113912 trainer/global_step‚2499ˆarÎ¥È ' lr-AdamW/pg1‚0.00019755751902113912 ' lr-AdamW/pg2‚0.00019755751902113912 trainer/global_step‚2499 % _timestamp‚1.7884305561497593e+09 _runtime‚2534.504783732 _step‚202ÊpÎþ~2¦- _wandbruntime‚2534  _runtime‚2534'3Òú'§ôO UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78955008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340065363259392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26854187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884305561505897e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5342399215616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340065363259392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚503 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345410894259712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5333358412781 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53857715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19627208 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5342399215616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339570011187181 trainer/global_step‚2499 _step‚203 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345410894259712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98739774 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44882059 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345407762475008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290570539299328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7418859066880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334236652774400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44882065 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340065389650944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297989319411200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290570511164928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44882059 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53880820 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339573140084069 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334236678202368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5345504608768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53857715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297989398366208 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340065389650944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7418808246272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340065363259392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚503 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345410894259712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5333358412781 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53857715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19627208 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5342399215616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339570011187181 trainer/global_step‚2499 _step‚203 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345410894259712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98739774 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44882059 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345407762475008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290570539299328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7418859066880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334236652774400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44882065 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340065389650944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297989319411200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290570511164928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44882059 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53880820 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339573140084069 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334236678202368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5345504608768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53857715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297989398366208 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340065389650944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7418808246272 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53878805 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334366075203584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345541368877056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 % _timestamp‚1.7884305570594275e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98778148 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345544500675072 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5347616935936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340196883739136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290682950163456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5335467205881 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19634901 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46498834 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340196883739136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340196857347584 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98801285 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44899343 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298104780957696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44899343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339701516981497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345544500675072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334366049775616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98801285 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 6Þ¼ '­ƒN REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44899349 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚204 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98778148 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7421830794240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46498834 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340196883739136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340196857347584 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98801285 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44899343 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298104780957696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44899343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339701516981497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345544500675072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334366049775616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98801285 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5347616935936 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7421779986944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53878805 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334366075203584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345541368877056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 % _timestamp‚1.7884305570594275e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98778148 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345544500675072 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5347616935936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340196883739136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290682950163456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5335467205881 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19634901 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚346640067668992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 % _timestamp‚1.7884306030953524e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54915567 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5451134715392 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47380175 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346145314494189 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5451134715392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45746259 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚346640067668992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54915567 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20013338 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5448029322240 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54938672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100661826 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 JDDeviceStatsMZ2þ–onitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340706528263168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5448029322240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27366837 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352088096991232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20013866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296191054371328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352088096991232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352091228775936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303758020198400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100684937 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45746265bÚ(¼ýO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚346640094060544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303758103347712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45746259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296191082505728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7566965827072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚207 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚346640094060544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27366840 trainer/global_step‚2549 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352088096991232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352091228775936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303758020198400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100684937 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45746265 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346148443391077 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5438811658989 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45746265 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340706502835200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚346640067668992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 % _timestamp‚1.7884306030953524e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54915567 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5451134715392 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47380175 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346145314494189 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5451134715392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45746259 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚346640067668992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54915567 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20013338 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5448029322240 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54938672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100661826 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340706528263168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5448029322240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27366837 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352088096991232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20013866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296191054371328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340835899836416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27377093 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45763543 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20020901 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346279949185601 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5453247042560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100723337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303873469872640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚340835925264384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5440920452089 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45763543 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296303493369856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47398519 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47397991 % _timestamp‚1.7884306039930756e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 trainer/global_step‚2549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352221703393280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54936657 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45763549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352224835191296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27377090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚346771561757184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚346771588148736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303873386736640þ <0™¿†N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5450141636096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45763543 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20020901 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346279949185601 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5453247042560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100723337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303873469872640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚340835925264384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5440920452089 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45763543 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296303493369856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47398519 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47397991 % _timestamp‚1.7884306039930756e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 trainer/global_step‚2549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352221703393280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54936657 H@DeviceStatsMonitor.on_train_batch_end/inactn}G sive_split.all.current‚528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45763549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352224835191296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27377090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚346771561757184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚346771588148736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303873386736640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7569976502784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5444023921217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296303465235456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100700200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚346771561757184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54959788 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100700200 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚208 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340835899836416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27377093 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48280944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353214772078592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5553659428864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20400945 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20401454 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358771563292160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353214772078592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5553659428864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358768431507456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46610465 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55973419 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7715242228736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2628.677609128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55973419 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353214798470144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _step‚211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 % _timestamp‚1.7884306503222787e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5556764822016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48280432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309526918761472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358771563292160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27879487 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27879490 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102606989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352723746698085 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5556764822016 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358768431507456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46610465 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55973419 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7715242228736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2628.677609128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55973419 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353214798470144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _step‚211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 % _timestamp‚1.7884306503222787e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5556764822016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48280432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309526918761472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358771563292160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27879487 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27879490 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102606989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352723746698085 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5556764822016 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48280944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353214772078592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5553659428864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20400945 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20401454 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358771563292160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 \HDeviceStatsMonitor.Ò±œð¨on_train_batch_start/allocated_bytes.large_pool.freed‚353214772078592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5553659428864Ó'’<À2Ï- _wandbruntime‚2629  _runtime‚2629!è· 'ÐN PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46627749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353346266166784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27889743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309642296051712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20409017 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7718210765312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358902037909504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102645389 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46627749 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56017640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352855252492609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚ 2629.56922375 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46627743 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5558877149184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353346292558336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301924036576256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358905169707520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5558877149184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56017640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309642219207168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48710144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5549477167425 % _timestamp‚1.7884306512137856e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347305775325184 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5555771742720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚490 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48298267 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55994509 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016[SN 'цN XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309642219207168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48710144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5549477167425 % _timestamp‚1.7884306512137856e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347305775325184 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5555771742720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚490 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48298267 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55994509 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46627749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353346266166784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27889743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309642296051712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20409017 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7718210765312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358902037909504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102645389 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46627749 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56017640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352855252492609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚ 2629.56922375 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46627743 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5558877149184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353346292558336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301924036576256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358905169707520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5558877149184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56017640ÔغÏ&2Ò- _wandbruntime‚2629  _runtime‚2629YëåjVÓÐ 1loss/SRqqt2/assignment_loss‚2.073523218775618 2loss/SRqqt3/assignment_loss‚1.6867265723231766 .loss/FBt3/detection_loss‚2.486813416847816 % _timestamp‚1.7884306512153838e+09 _step‚213 /loss/FBt1/detection_loss‚1.2409430993004487 /loss/FRt1/assignment_loss‚2.309456403913169 0loss/FRt3/assignment_loss‚2.3210994056292944 2loss/SRqqt4/assignment_loss‚1.9768988794336717 1loss/FBt1/assignment_loss‚0.27062199439889506 1loss/FBt3/assignment_loss‚0.35675851377443624 .loss/FRt4/detection_loss‚3.039805546402931 1loss/SRqqt3/detection_loss‚1.2543144038981862 trainer/global_step‚2599 _runtime‚2629.570378239 /loss/FRt2/detection_loss‚2.6699849351858482 1loss/SRqqt4/detection_loss‚2.7296399821837745 .loss/FBt4/detection_loss‚1.580622061087591 &loss/total_loss‚49.318628622129765 /loss/FRt4/assignment_loss‚5.223380943139394 0loss/FBt2/assignment_loss‚0.5112907350121527 .loss/FBt2/detection_loss‚2.504443272948265 epoch‚2 0loss/SRqqt2/detection_loss‚2.745174581634587 .loss/FRt1/detection_loss‚1.918540553286158 0loss/SRqqt1/detection_loss‚1.328701381966219 /loss/FRt2/assignment_loss‚5.441346009572347 0loss/FBt4/assignment_loss‚0.3331307659671261 .loss/FRt3/detection_loss‚1.852568761578628 2loss/SRqqt1/assignment_loss‚1.4628431838700326".[ÔÕ /loss/FRt2/detection_loss‚2.6699849351858482 1loss/SRqqt4/detection_loss‚2.7296399821837745 .loss/FBt4/detection_loss‚1.580622061087591 &loss/total_loss‚49.318628622129765 /loss/FRt4/assignment_loss‚5.223380943139394 0loss/FBt2/assignment_loss‚0.5112907350121527 .loss/FBt2/detection_loss‚2.504443272948265 epoch‚2 0loss/SRqqt2/detection_loss‚2.745174581634587 .loss/FRt1/detection_loss‚1.918540553286158 0loss/SRqqt1/detection_loss‚1.328701381966219 /loss/FRt2/assignment_loss‚5.441346009572347 0loss/FBt4/assignment_loss‚0.3331307659671261 .loss/FRt3/detection_loss‚1.852568761578628 2loss/SRqqt1/assignment_loss‚1.4628431838700326 1loss/SRqqt2/assignment_loss‚2.073523218775618 2loss/SRqqt3/assignment_loss‚1.6867265723231766 .loss/FBt3/detection_loss‚2.486813416847816 % _timestamp‚1.7884306512153838e+09 _step‚213 /loss/FBt1/detection_loss‚1.2409430993004487 /loss/FRt1/assignment_loss‚2.309456403913169 0loss/FRt3/assignment_loss‚2.3210994056292944 2loss/SRqqt4/assignment_loss‚1.9768988794336717 1loss/FBt1/assignment_loss‚0.27062199439889506 1loss/FBt3/assignment_loss‚0.35675851377443624 .loss/FRt4/detection_loss‚3.039805546402931 1loss/SRqqt3/detection_loss‚1.2543144038981862 trainer/global_step‚2599 _runtime‚2629.570378239Õ &Ûç“Õ‚(:ˆ Å’åÔþŽ·µ gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚144.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÑYÖ‚(:Î Å’åÔIJú¶& disk.nvme0n1.out‚21626.80810546875 network.sent‚ 5.600586e+06# disk.dm-0.out‚21626.80810546875 disk.sda.in‚0) proc.memory.availableMB‚ 4473.05078125 disk.drbd1716.in‚0 disk.dm-1.out‚ 96.453125( disk.nvme0c0n1.out‚21626.80810546875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0$ memory_percent‚72.69866466522217 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.drbd2080.in‚0% disk./.usageGB‚274.28332901000977 disk.sda3.out‚ 96.453125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 162.18359375# proc.memory.rssMB‚ 3040.46484375) proc.memory.percent‚18.55752468109131 disk.dm-0.in‚ 162.18359375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.269417891986237 network.recv‚947918* disk./.usagePercent‚18.407384348903616 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 96.453125 disk.nvme0n1.in‚ 162.18359375ê %’ׂ(:‡ Ô’åÔüæ· gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚299.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªãÈçVØ‚(:Ë Ô’åÔùÿù¸ disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0) proc.memory.percent‚18.55752468109131 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0$ memory_percent‚72.69570827484131) proc.memory.availableMB‚ 4473.53515625) disk./.usagePercent‚18.40739612525019 disk.dm-1.out‚ 96.859375 disk.sda1.in‚0 network.recv‚953305 disk.rbd1.out‚ 1.533203125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 162.18359375 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚21680.22802734375 disk.dm-0.in‚ 162.18359375# proc.memory.rssMB‚ 3040.46484375# disk./.usageGB‚274.283504486084& disk.nvme0n1.out‚21680.22802734375 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 96.859375 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0 cpu‚25.268150674754963 network.sent‚ 5.708595e+06# disk.dm-0.out‚21680.22802734375 disk.sda.out‚ 96.859375 disk.drbd2265.out‚0Ûf™q“Ù‚(:ˆ ã’åÔå¡Êµ gpu.0.gpu‚76.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.742' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.5484 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í!ÝUÚ‚(:Ê ã’åÔÛ‹»· disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3040.46484375 network.recv‚958003" disk.nvme0c0n1.in‚ 162.18359375 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.18359375$ memory_percent‚72.69878387451172 proc.cpu.threads‚17! disk.dm-0.out‚21733.345703125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 162.18359375 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 97.45703125 network.sent‚ 5.714523e+06 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda.in‚0$ disk.nvme0n1.out‚21733.345703125 disk.dm-1.in‚0 cpu‚25.266873938999698$ disk./.usageGB‚274.3028144836426 disk.sda3.out‚ 97.45703125 disk.drbd2265.in‚0) proc.memory.percent‚18.55752468109131* disk./.usagePercent‚18.408692035388516 disk.drbd2080.out‚0 disk.dm-1.out‚ 97.45703125 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚21733.345703125 disk.sda2.in‚0& proc.memory.availableMB‚ 4473.03125uú¤"2Û- _wandbruntime‚2675  _runtime‚26756[!“ÉÜà _runtime‚2675.850696132 _step‚214 ' lr-AdamW/pg1‚0.00019585667798664754 ' lr-AdamW/pg2‚0.00019585667798664754 trainer/global_step‚2649 % _timestamp‚1.7884306974956625e+09n@Ô`ÎÝÈ ' lr-AdamW/pg1‚0.00019585667798664754 ' lr-AdamW/pg2‚0.00019585667798664754 trainer/global_step‚2649 % _timestamp‚1.7884306974956625e+09 _runtime‚2675.850696132 _step‚214Öécý2Þ- _wandbruntime‚2675  _runtime‚2675ݲC¤ûßøO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365451897808384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47474665 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57031271 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57054376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307432140784128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47474659 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315295740758528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315295819713536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57031271 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365448766023680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5662394928640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49181386 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365448766023680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20789764 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28392137 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.smaàíæµll_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47474665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20789249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359295921108205 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28392140 % _timestamp‚1.7884306974964836e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359789476488192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359789502879744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7863650795008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47474659 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57054376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552TtÛ‡(àýO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5659289535488 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78955008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5659289535488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104529041 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49181904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5652821620325 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104505930 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307432168918528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353646202956800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359299050005093 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5649718151405 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚359789476488192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 _step‚215 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365451897808384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚359789502879744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365451897808384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47474665 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57031271 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57054376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307432140784128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47474659 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315295740758528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315295819713536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57031271 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365448766023680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5662394928640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49181386 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365448766023680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20789764 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28392137 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47474665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20789249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359295921108205 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28392140 % _timestamp‚1.7884306974964836e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359789476488192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359789502879744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7863650795008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47474659 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57054376 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47491949 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚484 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5664507255808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5651826944505 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104544304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365582372425728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49199411 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47491943 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359427426902521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20797021 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315411195693056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359920996967936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359920970576384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104544304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28402390 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20797505 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353775625385984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5661401849344 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359920996967936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 % _timestamp‚1.7884306983894484e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57052361 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5661401849344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307544551648256 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359920970576384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚2676.745121479 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104567441 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50807296 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365585504223744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57052361 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104567441 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7866565103104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359430555799617 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359920996967936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 % _timestamp‚1.7884306983894484e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57052361 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5661401849344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307544551648256 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359920970576384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚2676.745121479 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104567441 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50807296 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365585504223744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57052361 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104567441 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7866565103104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359430555799617 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47491949 OGDeviceStatsMonitor.on_train_batcö)1p h_end/inactive_split.small_pool.current‚484 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5664507255808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5651826944505 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104544304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365582372425728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49199411 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47491943 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359427426902521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20797021 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315411195693056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180Ø&(8Ý2ä- _wandbruntime‚2676  _runtime‚2676iHjZåÔ 0loss/FBt1/assignment_loss‚0.3285751060330976 1loss/SRqqt1/detection_loss‚1.4505149610896608 1loss/SRqqt2/detection_loss‚3.0355531752109526 trainer/global_step‚2649 2loss/SRqqt4/assignment_loss‚1.6514791631450256 1loss/SRqqt3/detection_loss‚1.2913152117554734 epoch‚2 0loss/FRt1/assignment_loss‚2.2057691184537753 /loss/FRt2/assignment_loss‚4.771009378433227 1loss/SRqqt3/assignment_loss‚1.131457678746523 /loss/FRt2/detection_loss‚2.7779221296310426 .loss/FBt1/detection_loss‚0.936584725851814 2loss/SRqqt1/assignment_loss‚1.5400912068617434 /loss/FBt3/detection_loss‚2.3876682290664086 _runtime‚2676.746320743 0loss/FRt3/assignment_loss‚2.1969740334898233 1loss/SRqqt2/assignment_loss‚2.419487679004669 /loss/FRt3/detection_loss‚1.9032857061829418 1loss/SRqqt4/detection_loss‚2.5614824811617534 1loss/FBt2/assignment_loss‚0.43987990315487163 0loss/FBt4/assignment_loss‚0.5018732474206693 .loss/FRt4/detection_loss‚3.172643999258677 /loss/FBt4/detection_loss‚1.8810410926287824 %loss/total_loss‚48.19524095311641 /loss/FRt4/assignment_loss‚4.765511592229207 % _timestamp‚1.7884306983913674e+09 1loss/FBt3/assignment_loss‚0.18560065642192897 .loss/FRt1/detection_loss‚1.984155811369419 /loss/FBt2/detection_loss‚2.6753646665149264 _step‚217¡Ü’_æÙ % _timestamp‚1.7884306983913674e+09 1loss/FBt3/assignment_loss‚0.18560065642192897 .loss/FRt1/detection_loss‚1.984155811369419 /loss/FBt2/detection_loss‚2.6753646665149264 _step‚217 0loss/FBt1/assignment_loss‚0.3285751060330976 1loss/SRqqt1/detection_loss‚1.4505149610896608 1loss/SRqqt2/detection_loss‚3.0355531752109526 trainer/global_step‚2649 2loss/SRqqt4/assignment_loss‚1.6514791631450256 1loss/SRqqt3/detection_loss‚1.2913152117554734 epoch‚2 0loss/FRt1/assignment_loss‚2.2057691184537753 /loss/FRt2/assignment_loss‚4.771009378433227 1loss/SRqqt3/assignment_loss‚1.131457678746523 /loss/FRt2/detection_loss‚2.7779221296310426 .loss/FBt1/detection_loss‚0.936584725851814 2loss/SRqqt1/assignment_loss‚1.5400912068617434 /loss/FBt3/detection_loss‚2.3876682290664086 _runtime‚2676.746320743 0loss/FRt3/assignment_loss‚2.1969740334898233 1loss/SRqqt2/assignment_loss‚2.419487679004669 /loss/FRt3/detection_loss‚1.9032857061829418 1loss/SRqqt4/detection_loss‚2.5614824811617534 1loss/FBt2/assignment_loss‚0.43987990315487163 0loss/FBt4/assignment_loss‚0.5018732474206693 .loss/FRt4/detection_loss‚3.172643999258677 /loss/FBt4/detection_loss‚1.8810410926287824 %loss/total_loss‚48.19524095311641 /loss/FRt4/assignment_loss‚4.765511592229207ÙEº/“ç‚(:ˆ ò’åÔŠŒø¶ gpu.0.gpu‚63.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚266.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÀ†Oè‚(:Ä ò’åÔ«Õ©· disk.dm-0.in‚ 162.18359375 disk.sda.out‚ 97.8515625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.18359375! disk.dm-0.out‚21776.970703125 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 1.533203125$ memory_percent‚79.93535995483398 disk.dm-1.in‚0 disk.dm-1.out‚ 97.8515625 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17' disk./.usagePercent‚18.409035853507 disk.drbd1716.in‚0) proc.memory.percent‚18.55752468109131$ disk./.usageGB‚274.3079376220703 disk.sda3.out‚ 97.8515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚21776.970703125 disk.sda2.out‚0' proc.memory.availableMB‚ 3287.390625# proc.memory.rssMB‚ 3040.46484375 disk.drbd2266.out‚0 network.sent‚ 5.720481e+06 network.recv‚962022 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda1.out‚0& disk.nvme0c0n1.out‚21776.970703125 cpu‚25.26567619567609ÂÏ¢v“é‚(:ˆ “åÔ’ïñ´ gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚296.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kUFê‚(:» “åÔ¿¸Ô· disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 162.18359375# proc.memory.rssMB‚ 3040.46484375) proc.memory.percent‚18.55752468109131 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.sda.in‚0$ disk./.usageGB‚274.3134078979492 disk.drbd2080.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.18359375$ memory_percent‚92.77141094207764 disk.drbd2080.in‚0 disk.sda.out‚98.5625# disk.nvme0n1.out‚21838.44140625 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.out‚98.5625 disk.dm-0.out‚21838.44140625 cpu‚25.26448296432225 disk.drbd2265.out‚0 network.sent‚ 5.829503e+06 network.recv‚968548( disk./.usagePercent‚18.4094029683111 disk.dm-1.in‚0 disk.sda3.out‚98.5625 disk.drbd1716.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚21838.44140625) proc.memory.availableMB‚ 1184.33203125 proc.cpu.threads‚17 disk.sda1.in‚0ÂÒWø’ë‚(:‡ “åÔóœµ gpu.0.gpu‚89.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚279.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.933 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ÈSì‚(:È “åÔ¼Ç˶ disk.dm-0.in‚ 162.18359375 disk.sda3.out‚ 98.95703125 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1716.in‚0 disk.sda.out‚ 98.95703125% disk./.usageGB‚274.21961212158203 disk.dm-1.out‚ 98.95703125 disk.sda1.in‚0 disk.sda.in‚0" disk.nvme0n1.out‚ 21880.3203125" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.out‚ 21880.3203125 disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 3040.46484375 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0$ memory_percent‚92.77160167694092) proc.memory.availableMB‚ 1184.30078125 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 21880.3203125 disk.sda3.in‚0 disk.drbd1716.out‚0 network.recv‚972124 disk.rbd0.out‚0 network.sent‚ 5.834783e+06* disk./.usagePercent‚18.403108255059202 disk.sda2.in‚0 disk.drbd2266.in‚0) proc.memory.percent‚18.55752468109131 cpu‚25.263238337671662 proc.cpu.threads‚17,3ÙN2í- _wandbruntime‚2723  _runtime‚2723ÞœÉîà ' lr-AdamW/pg1‚0.00019525056903439963 ' lr-AdamW/pg2‚0.00019525056903439963 trainer/global_step‚2699 % _timestamp‚1.7884307450249457e+09 _runtime‚2723.379978135 _step‚218Šu¹JÎïÈ % _timestamp‚1.7884307450249457e+09 _runtime‚2723.379978135 _step‚218 ' lr-AdamW/pg1‚0.00019525056903439963 ' lr-AdamW/pg2‚0.00019525056903439963 trainer/global_step‚2699ÚÓ*¨2ð- _wandbruntime‚2723  _runtime‚2723¤†§þ'ñøO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚321063851020288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48338865 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚360116078445568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5755171397613 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 trainer/global_step‚2699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366364207289344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106427982 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366364180897792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58112228 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365874353312101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58089123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58089123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28904787 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21175527 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50080314 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8011138895360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5768025035264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366364180897792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5764919642112 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚2723.380870319 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884307450256658e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚372132232324608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366364207289344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313052683990528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48338865 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5758274866533 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372132232324608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372129100539904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21176072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313052712124928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372129100539904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321063769968128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984õö(òýO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48338865 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚360116078445568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5755171397613 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 trainer/global_step‚2699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366364207289344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106427982 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366364180897792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58112228 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365874353312101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58089123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58089123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28904787 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21175527 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50080314 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8011138895360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5768025035264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366364180897792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5764919642112 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚2723.380870319 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884307450256658e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚372132232324608 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366364207289344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313052683990528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48338865 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5758274866533 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372132232324608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372129100539904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21176072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313052712124928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372129100539904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321063769968128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚321063851020288Û—_“2ó- _wandbruntime‚2724  _runtime‚2724¹z©ÅYô‚N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚514 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313165122989056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28915040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 B·¯"8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2724.272537205 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366495674985984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366495674985984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106466356 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8014055269888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5767031955968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21183242 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5757280190713 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372265838739968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5770137362432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28915043 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366002730209529 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106489493 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5770137362432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48356143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58110213 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58133344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21183756 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313165094854656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52904448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366005859106625 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106466356 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360245475446784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48356149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106489493 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48356149 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5760383659841 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58133344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372262706941952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366495701377536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5767031955968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8014108174336 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50098799 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50098282 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48356143 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464‘éj 'õ‡N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106466356 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8014055269888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5767031955968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21183242 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5757280190713 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372265838739968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5770137362432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28915043 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366002730209529 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106489493 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5770137362432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48356143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58110213 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58133344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21183756 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313165094854656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52904448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366005859106625 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106466356 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360245475446784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48356149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106489493 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48356149 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5760383659841 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58133344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372262706941952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366495701377536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5767031955968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8014108174336 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50098799 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50098282 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48356143 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚514 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313165122989056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28915040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2724.272537205 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366495674985984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366495674985984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302Ü Æ8B2ö- _wandbruntime‚2724  _runtime‚2724µô§W÷Ñ -loss/FRt1/detection_loss‚1.70756265964057 epoch‚2 /loss/FRt4/assignment_loss‚4.644414935793195 1loss/SRqqt2/detection_loss‚2.8937432205677034 0loss/FRt1/assignment_loss‚2.3925435824974164 2loss/SRqqt1/assignment_loss‚1.4066007886000949 %loss/total_loss‚50.75355528930933 2loss/SRqqt4/assignment_loss‚2.1784455654521784 1loss/FBt3/assignment_loss‚0.15602432405704839 0loss/FBt1/assignment_loss‚0.1916083149154881 /loss/FBt1/detection_loss‚1.0198877280799008 0loss/FBt2/assignment_loss‚0.5022813379764557 /loss/FBt4/detection_loss‚1.4745579182337492 trainer/global_step‚2699 % _timestamp‚1.7884307459185085e+09 /loss/FRt3/assignment_loss‚3.128305913403977 1loss/SRqqt3/assignment_loss‚1.280307966941858 /loss/FRt3/detection_loss‚2.3466888063175735 1loss/SRqqt3/detection_loss‚1.5434620402315085 0loss/SRqqt4/detection_loss‚2.549368319908778 /loss/FBt3/detection_loss‚2.6901681377337527 _step‚221 1loss/FBt4/assignment_loss‚0.45026323937166196 -loss/FRt2/detection_loss‚2.75235163503223 /loss/FRt4/detection_loss‚3.2494043622698103 1loss/SRqqt1/detection_loss‚1.2300635593524878 /loss/FBt2/detection_loss‚3.2710716843605043 _runtime‚2724.273468222 /loss/FRt2/assignment_loss‚5.212008595466614 1loss/SRqqt2/assignment_loss‚2.482420653104782X® Ç\øÖ /loss/FRt3/assignment_loss‚3.128305913403977 1loss/SRqqt3/assignment_loss‚1.280307966941858 /loss/FRt3/detection_loss‚2.3466888063175735 1loss/SRqqt3/detection_loss‚1.5434620402315085 0loss/SRqqt4/detection_loss‚2.549368319908778 /loss/FBt3/detection_loss‚2.6901681377337527 _step‚221 1loss/FBt4/assignment_loss‚0.45026323937166196 -loss/FRt2/detection_loss‚2.75235163503223 /loss/FRt4/detection_loss‚3.2494043622698103 1loss/SRqqt1/detection_loss‚1.2300635593524878 /loss/FBt2/detection_loss‚3.2710716843605043 _runtime‚2724.273468222 /loss/FRt2/assignment_loss‚5.212008595466614 1loss/SRqqt2/assignment_loss‚2.482420653104782 -loss/FRt1/detection_loss‚1.70756265964057 epoch‚2 /loss/FRt4/assignment_loss‚4.644414935793195 1loss/SRqqt2/detection_loss‚2.8937432205677034 0loss/FRt1/assignment_loss‚2.3925435824974164 2loss/SRqqt1/assignment_loss‚1.4066007886000949 %loss/total_loss‚50.75355528930933 2loss/SRqqt4/assignment_loss‚2.1784455654521784 1loss/FBt3/assignment_loss‚0.15602432405704839 0loss/FBt1/assignment_loss‚0.1916083149154881 /loss/FBt1/detection_loss‚1.0198877280799008 0loss/FBt2/assignment_loss‚0.5022813379764557 /loss/FBt4/detection_loss‚1.4745579182337492 trainer/global_step‚2699 % _timestamp‚1.7884307459185085e+09ÝÂXÝù‚(:’ Ÿ“åÔÜúí´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚288.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.69740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶CRú‚(:Ç Ÿ“åÔÝäᶠdisk.drbd2080.in‚0 network.sent‚ 5.840761e+06 disk.dm-1.out‚ 99.5546875 disk.rbd1.out‚ 1.533203125" disk.nvme0c0n1.in‚ 162.18359375 disk.sda.in‚0 disk.sda.out‚ 99.5546875# memory_percent‚92.0034408569336 network.recv‚976550% disk./.usageGB‚274.24991607666016 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0& proc.memory.availableMB‚ 1310.15625* disk./.usagePercent‚18.405141978911203 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 3040.46484375) proc.memory.percent‚18.55752468109131 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.18359375 proc.cpu.threads‚17! disk.dm-0.out‚21940.396484375 disk.sda3.out‚ 99.5546875 disk.drbd2266.in‚0 disk.dm-0.in‚ 162.18359375 disk.drbd2265.out‚0$ disk.nvme0n1.out‚21940.396484375 cpu‚25.262074291612837 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚21940.396484375 disk.sda3.in‚0ó:”z“û‚(:ˆ ®“åÔøÊ›µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÕDZü‚(:Ï ®“åÔðã̶" disk.nvme0c0n1.in‚ 162.18359375& disk.nvme0c0n1.out‚21968.693359375 network.sent‚ 5.949333e+06 disk.dm-1.out‚ 100.24609375$ disk./.usageGB‚274.2583656311035 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 100.24609375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 network.recv‚982980 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1310.40234375) proc.memory.percent‚18.55752468109131* disk./.usagePercent‚18.405709035599546 disk.sda1.out‚0 disk.sda.out‚ 100.24609375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 3040.46484375 disk.sda.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚25.26092928713323 proc.cpu.threads‚17$ disk.nvme0n1.out‚21968.693359375 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 162.18359375! disk.dm-0.out‚21968.693359375$ memory_percent‚92.00193881988525 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 0Pý‚(:’ ½“åÔ„âÕ´ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚285.334' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.06679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ù¨]_þ‚(:Ô ½“åÔþø · disk.sda.in‚0 disk.sda.out‚ 100.6796875& disk.nvme0n1.out‚22013.98876953125 network.recv‚987590 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 162.18359375# disk.dm-0.out‚22013.98876953125 disk.drbd1716.in‚0$ memory_percent‚92.00341701507568 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1310.16015625 cpu‚25.259871167542116 network.sent‚ 5.955646e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 3040.46484375 disk.sda2.out‚0 disk.dm-1.out‚ 100.6796875 disk.sda3.out‚ 100.6796875 disk.drbd2080.in‚0 disk.drbd2080.out‚0% disk./.usageGB‚274.27851486206055 disk.rbd1.in‚0( disk.nvme0c0n1.out‚22013.98876953125) proc.memory.percent‚18.55752468109131* disk./.usagePercent‚18.407061267395406 disk.rbd1.out‚ 1.533203125 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.dm-0.in‚ 162.18359375&ØSÕ2ÿ- _wandbruntime‚2770  _runtime‚2770®Š).ɀà _step‚222 ' lr-AdamW/pg1‚0.00019462508624135675 ' lr-AdamW/pg2‚0.00019462508624135675 trainer/global_step‚2749 % _timestamp‚1.7884307922991207e+09 _runtime‚2770.6541606941Å€ÎÈ _runtime‚2770.654160694 _step‚222 ' lr-AdamW/pg1‚0.00019462508624135675 ' lr-AdamW/pg2‚0.00019462508624135675 trainer/global_step‚2749 % _timestamp‚1.7884307922991207e+09Þ12‚- _wandbruntime‚2770  _runtime‚2770W;r[ƒøO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚366585928506368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚223 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378809435056128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378809435056128 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59170080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108373145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108350034 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326831809061376 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372938911698944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372938885307392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8158634782208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5870549748736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50979769 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5870549748736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21562329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318673255331328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49203059 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81052160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚378812566840832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3724465277222215è^/(„ýO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5870549748736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50979769 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5870549748736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21562329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318673255331328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49203059 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81052160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚378812566840832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372446527722221 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚366585928506368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚223 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378809435056128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378809435056128 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59170080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108373145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108350034 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326831809061376 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372938911698944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372938885307392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8158634782208ßʘ2…- _wandbruntime‚2771  _runtime‚2771Üž('†N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373070379395584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21569867 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372578033516537 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108388408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21569354 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788430793191204e+09 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49220343 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49220349 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59168065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326947173029376 _runtime‚2771.546711354 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108411545 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373070379395584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5865836906049 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52904448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5875767469056 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108388408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5875767469056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378946173256192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59191196 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 trainer/global_step‚2749 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318785666195456 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108411545 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326947254068224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59191196 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5872662062592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373070405787136 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29427693 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50997044 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373070405787136sºÃÌ¿‡†N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50997044 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373070405787136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373070379395584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21569867 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372578033516537 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108388408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21569354 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788430793191204e+09 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49220343 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49220349 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59168065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326947173029376 _runtime‚2771.546711354 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108411545 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373070379395584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5865836906049 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52904448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5875767469056 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108388408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5875767469056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378946173256192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59191196 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 trainer/global_step‚2749 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318785666195456 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108411545 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326947254068224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59191196 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5872662062592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373070405787136 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29427693 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0à˜>Væ2ˆ- _wandbruntime‚2771  _runtime‚2771©Fë&Z‰Ô /loss/FBt3/detection_loss‚2.2384088062323055 1loss/SRqqt3/assignment_loss‚1.151308325957507 0loss/SRqqt4/detection_loss‚2.743382066488266 /loss/SRqqt1/detection_loss‚1.52390326632828 0loss/FRt1/assignment_loss‚2.9388199132500272 2loss/SRqqt1/assignment_loss‚1.3911154217193606 1loss/FBt2/assignment_loss‚0.42684951301539925 1loss/FBt3/assignment_loss‚0.30887880858230854 0loss/FBt4/assignment_loss‚0.2704967896236061 /loss/FRt4/detection_loss‚3.4635255274019743 /loss/FBt1/detection_loss‚1.1763326445730722 /loss/FRt2/detection_loss‚2.4769184820113646 _runtime‚2771.548308379 /loss/FRt2/assignment_loss‚5.500209539167343 2loss/SRqqt2/assignment_loss‚2.6178928052677826 1loss/SRqqt2/detection_loss‚2.3574565368540146 epoch‚2 trainer/global_step‚2749 /loss/FRt4/assignment_loss‚5.602352506236026 1loss/SRqqt3/detection_loss‚1.3882864276878535 .loss/FBt2/detection_loss‚2.785789793187922 &loss/total_loss‚50.084740957399475 % _timestamp‚1.7884307931932878e+09 _step‚225 /loss/FRt3/assignment_loss‚2.411659481624762 /loss/FRt1/detection_loss‚1.8727281968476195 .loss/FRt3/detection_loss‚2.079866332312425 /loss/FBt4/detection_loss‚1.4767468586722587 2loss/SRqqt4/assignment_loss‚1.8171959024060655 1loss/FBt1/assignment_loss‚0.06461701195192751Î1ˆï_ŠÙ /loss/FRt2/assignment_loss‚5.500209539167343 2loss/SRqqt2/assignment_loss‚2.6178928052677826 1loss/SRqqt2/detection_loss‚2.3574565368540146 epoch‚2 trainer/global_step‚2749 /loss/FRt4/assignment_loss‚5.602352506236026 1loss/SRqqt3/detection_loss‚1.3882864276878535 .loss/FBt2/detection_loss‚2.785789793187922 &loss/total_loss‚50.084740957399475 % _timestamp‚1.7884307931932878e+09 _step‚225 /loss/FRt3/assignment_loss‚2.411659481624762 /loss/FRt1/detection_loss‚1.8727281968476195 .loss/FRt3/detection_loss‚2.079866332312425 /loss/FBt4/detection_loss‚1.4767468586722587 2loss/SRqqt4/assignment_loss‚1.8171959024060655 1loss/FBt1/assignment_loss‚0.06461701195192751 /loss/FBt3/detection_loss‚2.2384088062323055 1loss/SRqqt3/assignment_loss‚1.151308325957507 0loss/SRqqt4/detection_loss‚2.743382066488266 /loss/SRqqt1/detection_loss‚1.52390326632828 0loss/FRt1/assignment_loss‚2.9388199132500272 2loss/SRqqt1/assignment_loss‚1.3911154217193606 1loss/FBt2/assignment_loss‚0.42684951301539925 1loss/FBt3/assignment_loss‚0.30887880858230854 0loss/FBt4/assignment_loss‚0.2704967896236061 /loss/FRt4/detection_loss‚3.4635255274019743 /loss/FBt1/detection_loss‚1.1763326445730722 /loss/FRt2/detection_loss‚2.4769184820113646 _runtime‚2771.548308379á?î9i‹‚(:’ Ì“åÔ©ƒÄµ gpu.0.gpu‚68.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.324' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.66480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þZEaŒ‚(:Ö Ì“åÔɹ»· disk.drbd2266.in‚0 network.recv‚991061 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 101.17578125 disk.dm-1.in‚0 disk.dm-1.out‚ 101.17578125 disk.rbd1.out‚ 1.533203125* disk./.usagePercent‚18.406917391161162# disk.dm-0.out‚22059.78173828125 disk.drbd2265.in‚0& disk.nvme0n1.out‚22059.78173828125" disk.nvme0c0n1.in‚ 162.18359375( disk.nvme0c0n1.out‚22059.78173828125$ memory_percent‚92.00222492218018 cpu‚25.25865382133813 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 3040.46484375 disk.dm-0.in‚ 162.18359375 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0) proc.memory.percent‚18.55752468109131 proc.cpu.threads‚17 disk.sda3.out‚ 101.17578125 disk.drbd2266.out‚0% disk./.usageGB‚274.27637100219727 disk.sda2.in‚0) proc.memory.availableMB‚ 1310.35546875 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.18359375 network.sent‚ 5.960894e+06 disk.sda1.in‚0@e½_‚(:’ Û“åÔܘ•µ gpu.0.gpu‚68.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚287.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.510600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QâÊ÷\Ž‚(:Ñ Û“åÔ˪à· disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 162.18359375 disk.sda3.in‚0 disk.sda1.in‚0 cpu‚25.25762573785313 disk.sda.out‚ 101.77734375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.18359375# proc.memory.rssMB‚ 3040.46484375 network.recv‚997758 disk.drbd2265.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 1310.3359375% disk.nvme0n1.out‚22110.7080078125 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 6.069757e+06$ disk./.usageGB‚274.2945747375488 disk.sda2.in‚0 disk.dm-1.out‚ 101.77734375 disk.drbd2266.in‚0 disk.drbd1716.out‚0$ memory_percent‚92.00234413146973) proc.memory.percent‚18.55752468109131 disk.nvme0n1.in‚ 162.18359375' disk.nvme0c0n1.out‚22110.7080078125" disk.dm-0.out‚22110.7080078125 disk.sda3.out‚ 101.77734375 disk.sda1.out‚0* disk./.usagePercent‚18.408139059114557 disk.drbd2265.out‚0å@š›ž‚(:“ ê“åÔ”¥¯µ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚282.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.564800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†+ d‚(:Ù ê“åÔ÷ÕÛ¶ disk.sda.in‚0% disk.nvme0n1.out‚22155.4072265625 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 162.18359375$ memory_percent‚92.00236797332764) proc.memory.percent‚18.55752468109131 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚22155.4072265625" disk.dm-0.out‚22155.4072265625# proc.memory.rssMB‚ 3040.46484375 disk.sda.out‚ 102.39453125 network.recv‚ 1.002039e+06% disk./.usageGB‚274.30524826049805 disk.rbd0.in‚0 disk.sda3.out‚ 102.39453125 disk.dm-1.out‚ 102.39453125 disk.drbd2266.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1310.33203125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.18359375) disk./.usagePercent‚18.40885536819536 network.sent‚ 6.075616e+06 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 162.18359375 cpu‚25.256544546206488 disk.sda1.out‚0 disk.drbd2080.out‚05e·J2‘- _wandbruntime‚2817  _runtime‚2817 lˆüÆ’À % _timestamp‚1.7884308394742718e+09 _runtime‚ 2817.82929183 _step‚226 & lr-AdamW/pg1‚0.0001939803599571955 & lr-AdamW/pg2‚0.0001939803599571955 trainer/global_step‚2799kD/FË“Å trainer/global_step‚2799 % _timestamp‚1.7884308394742718e+09 _runtime‚ 2817.82929183 _step‚226 & lr-AdamW/pg1‚0.0001939803599571955 & lr-AdamW/pg2‚0.0001939803599571955â©ü•2”- _wandbruntime‚2817  _runtime‚2817²ˆ˜²þ'•øO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332599492697088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50067259 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385489769572352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373055778567168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29930090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 _runtime‚2817.830225974 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50067265 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60227932 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8305722293760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385489769572352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21943314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385492901357056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5979285248512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60227932 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379513589716992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5976179855360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324293770403328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379513616108544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50067265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324293798537728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379024959926117 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110272086 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5969181358949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110295197 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51873404 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373055753139200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5966077890029 % _timestamp‚1.7884308394749985e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60204827 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21942763 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8305777308672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385489769572352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21943314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385492901357056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5979285248512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60227932 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379513589716992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5976179855360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324293770403328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379513616108544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50067265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324293798537728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379024959926117 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110272086 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5969181358949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110295197 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51873404 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373055753139200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5966077890029 % _timestamp‚1.7884308394749985e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60204827 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21942763 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8305777308672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78843084036505e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50084549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50084543 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379645083805184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332714838515712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29940340 _step‚228 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50084543 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.smaFäÊÝll_pool.current‚522 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50084549 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60249048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324406209401856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110333597 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5968186683129 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60249048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978292169216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379153336823545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60225917 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚385623375974400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51890810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29940343 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5971290152257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110333597 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110310460 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51891335 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978292169216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440*4U` '™…N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110333597 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5968186683129 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60249048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978292169216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379153336823545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚2799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60225917 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚385623375974400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51890810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29940343 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5971290152257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110333597 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110310460 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51891335 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978292169216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78843084036505e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50084549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50084543 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379645083805184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332714838515712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29940340 _step‚228 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50084543 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50084549 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60249048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324406209401856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329914341744128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6071531136237 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50931459 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61285784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 $ _timestamp‚1.788430886409994e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22328107 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6074634605157 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚385597134336237 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6081809961984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚2864.765231504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386088320518144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚385600263233125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6081809961984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392173235873280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6084915355136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8452775304192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386088294126592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.l¯v peak‚28216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338367117048320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚379525603200000 trainer/global_step‚2849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50931465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30442740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8452720289280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392170104088576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392170104088576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61262679 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386088294126592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30442737 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚338367033899008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386088320518144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392173235873280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6084915355136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8452775304192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386088294126592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338367117048320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚379525603200000 trainer/global_step‚2849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50931465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30442740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8452720289280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392170104088576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392170104088576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61262679 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386088294126592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30442737 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚338367033899008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386088320518144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329914341744128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6071531136237 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50931459 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61285784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 $ _timestamp‚1.788430886409994e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22328107 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6074634605157 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚385597134336237 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6081809961984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚2864.765231504çHáÏž©‚(:“ ¦”åÔ£‘õ gpu.0.gpu‚55.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.615' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.723000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mîÃZ]ª‚(:Ò ¦”åÔÔཷ disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 104.78125 disk.dm-0.in‚ 162.18359375 disk.drbd2080.in‚0) proc.memory.percent‚18.55752468109131 proc.cpu.threads‚17 disk.drbd2266.in‚0$ disk./.usageGB‚274.3620910644531 disk.dm-1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚22334.09716796875 disk.sda1.out‚0$ memory_percent‚92.00434684753418 disk.rbd0.in‚0 disk.sda.out‚ 104.78125 cpu‚25.252243805426932 network.sent‚ 6.201793e+06 disk.sda2.in‚0 disk.sda3.out‚ 104.78125( proc.memory.availableMB‚ 1310.0078125# proc.memory.rssMB‚ 3040.46484375 disk.sda.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚18.412670136463074 disk.rbd0.out‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.drbd1716.out‚0 network.recv‚ 1.021423e+06& disk.nvme0n1.out‚22334.09716796875" disk.nvme0c0n1.in‚ 162.18359375# disk.dm-0.out‚22334.09716796875µ5F^2«- _wandbruntime‚2865  _runtime‚2865!S '¬N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61283769 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30452990 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112255649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30452993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6073639929337 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112232512 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52788690 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52789223 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6087027682304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330026724473856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6083922275840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112255649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61306900 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386219814606336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚385731769027649 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392303710490624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61283769 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50948743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392306842288640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386219814606336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚392306842288640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50948743 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50948743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392306842288640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386219814606336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚392306842288640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50948743 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112255649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30452993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6073639929337 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112232512 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52788690 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52789223 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6087027682304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330026724473856è)¬í2®- _wandbruntime‚2865  _runtime‚2865èÖöZ¯Ô /loss/FRt2/assignment_loss‚5.399737862440256 /loss/FRt3/detection_loss‚1.8943908885121346 0loss/FRt1/assignment_loss‚2.6602830862005553 .loss/FRt4/detection_loss‚3.079405050853203 /loss/SRqqt4/detection_loss‚2.70936640921761 /loss/FBt4/detection_loss‚1.5260247278313797 _step‚233 .loss/FBt1/detection_loss‚0.964617616230888 2loss/SRqqt2/assignment_loss‚1.7746362716987216 0loss/FBt4/assignment_loss‚0.3128829874365894 .loss/FRt2/detection_loss‚2.460473188987145 1loss/SRqqt1/detection_loss‚1.4824813111704223 &loss/total_loss‚48.911561852255545 trainer/global_step‚2849 2loss/SRqqt4/assignment_loss‚1.5436337636235882 /loss/FRt1/detection_loss‚1.8459311888242762 /loss/FBt2/detection_loss‚3.6264698108037314 1loss/FBt1/assignment_loss‚0.19039824861784765 1loss/FBt2/assignment_loss‚0.41471224930564354 1loss/SRqqt2/detection_loss‚2.6272360497507554 .loss/FBt3/detection_loss‚2.565529915002676 0loss/FRt3/assignment_loss‚2.5958545664946238 0loss/FRt4/assignment_loss‚4.5272268180189466 1loss/SRqqt3/detection_loss‚1.5588921868345804 epoch‚2 $ _timestamp‚1.788430887305086e+09 _runtime‚2865.660070138 2loss/SRqqt1/assignment_loss‚1.6152964450266896 2loss/SRqqt3/assignment_loss‚1.2056412788935833 0loss/FBt3/assignment_loss‚0.3304399304796999JÉe1_°Ù .loss/FBt3/detection_loss‚2.565529915002676 0loss/FRt3/assignment_loss‚2.5958545664946238 0loss/FRt4/assignment_loss‚4.5272268180189466 1loss/SRqqt3/detection_loss‚1.5588921868345804 epoch‚2 $ _timestamp‚1.788430887305086e+09 _runtime‚2865.660070138 2loss/SRqqt1/assignment_loss‚1.6152964450266896 2loss/SRqqt3/assignment_loss‚1.2056412788935833 0loss/FBt3/assignment_loss‚0.3304399304796999 /loss/FRt2/assignment_loss‚5.399737862440256 /loss/FRt3/detection_loss‚1.8943908885121346 0loss/FRt1/assignment_loss‚2.6602830862005553 .loss/FRt4/detection_loss‚3.079405050853203 /loss/SRqqt4/detection_loss‚2.70936640921761 /loss/FBt4/detection_loss‚1.5260247278313797 _step‚233 .loss/FBt1/detection_loss‚0.964617616230888 2loss/SRqqt2/assignment_loss‚1.7746362716987216 0loss/FBt4/assignment_loss‚0.3128829874365894 .loss/FRt2/detection_loss‚2.460473188987145 1loss/SRqqt1/detection_loss‚1.4824813111704223 &loss/total_loss‚48.911561852255545 trainer/global_step‚2849 2loss/SRqqt4/assignment_loss‚1.5436337636235882 /loss/FRt1/detection_loss‚1.8459311888242762 /loss/FBt2/detection_loss‚3.6264698108037314 1loss/FBt1/assignment_loss‚0.19039824861784765 1loss/FBt2/assignment_loss‚0.41471224930564354 1loss/SRqqt2/detection_loss‚2.6272360497507554é ÿÐ%’±‚(:‡ µ”åԿꣵ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚185.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~à,]²‚(:Ò µ”åÔ©ó¶ disk.dm-0.in‚ 162.18359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 network.sent‚ 6.299383e+06 disk.drbd2265.out‚0 disk.drbd2266.out‚0$ memory_percent‚92.00301170349121# proc.memory.rssMB‚ 3040.46484375) proc.memory.percent‚18.55752468109131 disk.rbd0.in‚0( proc.memory.availableMB‚ 1310.2265625 disk.sda.in‚0 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚22377.755859375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚18.41332730780347 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.25121012196863 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.out‚ 105.36328125 disk.dm-1.out‚ 105.36328125 disk.rbd1.out‚ 1.533203125! disk.dm-0.out‚22377.755859375 disk.drbd2080.out‚0 network.recv‚ 1.027999e+06# disk./.usageGB‚274.371883392334$ disk.nvme0n1.out‚22377.755859375 disk.sda3.out‚ 105.36328125 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 162.18359375œHÁº“³‚(:ˆ Ä”åÔ¥¡û´ gpu.0.gpu‚81.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚297.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ðD`´‚(:Õ Ä”åÔÖææ· disk.drbd2265.in‚0 disk.dm-1.out‚ 105.73828125 disk.sda3.in‚0 disk.sda3.out‚ 105.73828125 disk.sda.out‚ 105.73828125# proc.memory.rssMB‚ 3040.46484375 cpu‚25.25016064483642 network.recv‚ 1.032937e+06 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 162.18359375$ disk.nvme0n1.out‚22418.857421875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0* disk./.usagePercent‚18.415967257496558 disk.rbd0.in‚0! disk.dm-0.out‚22418.857421875) proc.memory.percent‚18.55752468109131 network.sent‚ 6.396872e+06 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1309.72265625$ disk./.usageGB‚274.4112205505371 disk.sda2.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚22418.857421875 disk.dm-0.in‚ 162.18359375 disk.sda1.in‚0$ memory_percent‚92.00608730316162 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 162.18359375íðr“µ‚(:ˆ Ó”åÔõŽµ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚283.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02"­àa¶‚(:Ö Ó”åÔóÛ¶¶% disk./.usageGB‚274.42161560058594 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375 disk.sda3.out‚ 106.4765625 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 cpu‚25.249077277352463 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 network.sent‚ 6.402968e+06 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.out‚ 106.4765625 disk.drbd2266.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚22463.0400390625 disk.sda2.out‚0' disk.nvme0c0n1.out‚22463.0400390625) proc.memory.percent‚18.55752468109131 disk.rbd0.in‚0 disk.sda.out‚ 106.4765625 network.recv‚ 1.03747e+06* disk./.usagePercent‚18.416664878027362 disk.drbd1716.in‚0$ memory_percent‚92.00465679168701# proc.memory.rssMB‚ 3040.46484375 disk.drbd2265.out‚0 disk.rbd1.out‚ 1.533203125" disk.dm-0.out‚22463.0400390625) proc.memory.availableMB‚ 1309.95703125 disk.sda.in‚0q!N2·- _wandbruntime‚2911  _runtime‚2911\³É¸Ã % _timestamp‚1.7884309335818028e+09 _runtime‚2911.936804099 _step‚234 ' lr-AdamW/pg1‚0.00019263371833779717 ' lr-AdamW/pg2‚0.00019263371833779717 trainer/global_step‚2899:N‹Î¹È ' lr-AdamW/pg1‚0.00019263371833779717 ' lr-AdamW/pg2‚0.00019263371833779717 trainer/global_step‚2899 % _timestamp‚1.7884309335818028e+09 _runtime‚2911.936804099 _step‚234êó÷m02º- _wandbruntime‚2911  _runtime‚2911!ö«1þ'»øO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392172437643245 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6176984382445 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6190545461760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62343636 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚2911.937772827 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51795659 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385995453260800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398853570389504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51795665 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62320531 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114116190 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385995478688768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51795665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22711553 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392175566540133 _step‚235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398850438604800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚335534884950528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884309335825183e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114139301 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344134643643392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6180087851365 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62343636 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53667478 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385995453260800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398853570389504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51795665 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62320531 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114116190 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385995478688768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51795665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22711553 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392175566540133 _step‚235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398850438604800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚335534884950528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884309335825183e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114139301 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344134643643392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6180087851365 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62343636 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53667478 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62343636 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚2911.937772827 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51795659 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_b-¥úõatch_end/inactive_split.large_pool.allocated‚30965643 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386124850262016 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62364752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8602707493376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53684944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398987176804864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51812943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51812949 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114154564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392794492624384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392303943437561 trainer/global_step‚2899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 _step‚236 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398987176804864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386124875689984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114177701 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335647267680256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6179093175545Õ¸(¯ '¿†N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335647267680256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6179093175545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344250003308032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392794492624384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182196644673 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 $ _timestamp‚1.788430934477085e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62341621 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30965643 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386124850262016 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62364752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8602707493376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53684944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398987176804864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51812943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51812949 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114154564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392794492624384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392303943437561 trainer/global_step‚2899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 _step‚236 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398987176804864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386124875689984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114177701ì Ç62À- _wandbruntime‚2912  _runtime‚2912g’œ‚ZÁÔ _step‚237 /loss/FBt3/detection_loss‚2.2779861986637115 /loss/FRt4/detection_loss‚3.0977539636872033 .loss/FBt4/detection_loss‚1.409370319201396 _runtime‚2912.833932477 trainer/global_step‚2899 % _timestamp‚1.7884309344789453e+09 0loss/FRt3/assignment_loss‚3.0378845632076263 /loss/FRt4/assignment_loss‚5.369723525914279 2loss/SRqqt3/assignment_loss‚1.2855579083164532 /loss/FRt1/detection_loss‚1.7206286950087226 /loss/FBt1/detection_loss‚1.1586675528902561 %loss/total_loss‚52.43898615681357 0loss/FBt2/assignment_loss‚0.5607204809784889 1loss/FBt3/assignment_loss‚0.10913903400069103 1loss/SRqqt3/detection_loss‚1.5824290470282236 /loss/FBt2/detection_loss‚3.8962007388472557 /loss/FRt1/assignment_loss‚2.281223775567235 1loss/SRqqt1/assignment_loss‚1.592716452915494 /loss/FRt3/detection_loss‚2.4245486319065095 1loss/SRqqt2/detection_loss‚3.7790902961384165 /loss/FRt2/assignment_loss‚5.273791451607981 0loss/FBt4/assignment_loss‚0.2259008123365465 0loss/SRqqt4/detection_loss‚2.484633296728134 2loss/SRqqt2/assignment_loss‚2.4538108841939406 1loss/FBt1/assignment_loss‚0.20117611238932617 0loss/SRqqt1/detection_loss‚1.346085069383063 2loss/SRqqt4/assignment_loss‚2.1046082982245613 /loss/FRt2/detection_loss‚2.7653390476780553 epoch‚2b˜*_ÂÙ 2loss/SRqqt4/assignment_loss‚2.1046082982245613 /loss/FRt2/detection_loss‚2.7653390476780553 epoch‚2 _step‚237 /loss/FBt3/detection_loss‚2.2779861986637115 /loss/FRt4/detection_loss‚3.0977539636872033 .loss/FBt4/detection_loss‚1.409370319201396 _runtime‚2912.833932477 trainer/global_step‚2899 % _timestamp‚1.7884309344789453e+09 0loss/FRt3/assignment_loss‚3.0378845632076263 /loss/FRt4/assignment_loss‚5.369723525914279 2loss/SRqqt3/assignment_loss‚1.2855579083164532 /loss/FRt1/detection_loss‚1.7206286950087226 /loss/FBt1/detection_loss‚1.1586675528902561 %loss/total_loss‚52.43898615681357 0loss/FBt2/assignment_loss‚0.5607204809784889 1loss/FBt3/assignment_loss‚0.10913903400069103 1loss/SRqqt3/detection_loss‚1.5824290470282236 /loss/FBt2/detection_loss‚3.8962007388472557 /loss/FRt1/assignment_loss‚2.281223775567235 1loss/SRqqt1/assignment_loss‚1.592716452915494 /loss/FRt3/detection_loss‚2.4245486319065095 1loss/SRqqt2/detection_loss‚3.7790902961384165 /loss/FRt2/assignment_loss‚5.273791451607981 0loss/FBt4/assignment_loss‚0.2259008123365465 0loss/SRqqt4/detection_loss‚2.484633296728134 2loss/SRqqt2/assignment_loss‚2.4538108841939406 1loss/FBt1/assignment_loss‚0.20117611238932617 0loss/SRqqt1/detection_loss‚1.346085069383063íãQŠžÃ‚(:“ â”åÔɆµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž“‰â]Ä‚(:Ò â”åÔèÐì¶ disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3040.46484375 disk.dm-0.in‚ 162.18359375) proc.memory.percent‚18.55752468109131 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.drbd2266.out‚0 disk.sda.out‚ 106.828125 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 cpu‚25.248057749174638 network.recv‚ 1.040903e+06 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda3.out‚ 106.828125 disk.drbd2265.out‚0% disk.nvme0n1.out‚22507.4951171875 proc.cpu.threads‚17$ disk./.usageGB‚274.4235496520996' disk.nvme0c0n1.out‚22507.4951171875" disk.dm-0.out‚22507.4951171875$ memory_percent‚92.00327396392822) proc.memory.availableMB‚ 1310.18359375 network.sent‚ 6.408599e+06) disk./.usagePercent‚18.41679467384722 disk.dm-1.out‚ 106.828125 disk.rbd1.out‚ 1.533203125 disk.sda.in‚0 disk.drbd2265.in‚0 disk.sda2.in‚0ˆý›“Å‚(:ˆ ñ”åÔ™·¸µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.7116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚080d_Æ‚(:Ô ñ”åÔƒäâ¶$ disk./.usageGB‚274.4375114440918 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0) proc.memory.percent‚18.55752468109131" disk.nvme0c0n1.in‚ 162.18359375 disk.sda3.out‚ 107.4296875 disk.drbd2265.in‚0# proc.memory.rssMB‚ 3040.46484375 disk.sda.in‚0 network.recv‚ 1.047479e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚22562.9033203125" disk.dm-0.out‚22562.9033203125 disk.drbd2080.in‚0# memory_percent‚92.0048475265503) disk./.usagePercent‚18.41773166142254 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 107.4296875 network.sent‚ 6.516818e+06 disk.dm-1.out‚ 107.4296875 disk.rbd0.out‚0% disk.nvme0n1.out‚22562.9033203125 disk.sda2.out‚0) proc.memory.availableMB‚ 1309.92578125 proc.cpu.threads‚17 disk.dm-0.in‚ 162.18359375 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.18359375 cpu‚25.247057501136982ÏòÁ«“Ç‚(:ˆ €•åÔ°ñ‰µ gpu.0.gpu‚69.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«î^aÈ‚(:Ö €•åÔá†ú¶ disk.sda3.out‚ 107.7578125 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1310.16796875# proc.memory.rssMB‚ 3040.46484375 disk.dm-0.in‚ 162.18359375 network.sent‚ 6.522084e+06$ disk./.usageGB‚274.4477996826172 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 162.18359375% disk.nvme0n1.out‚22592.4501953125 cpu‚25.246062485383177 network.recv‚ 1.051051e+06 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 107.7578125 disk.rbd0.in‚0 disk.dm-1.out‚ 107.7578125* disk./.usagePercent‚18.418422113742388" disk.dm-0.out‚22592.4501953125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0$ memory_percent‚92.00336933135986' disk.nvme0c0n1.out‚22592.4501953125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125 disk.drbd2265.in‚0) proc.memory.percent‚18.55752468109131I¼â2É- _wandbruntime‚2959  _runtime‚2959æ{yÈÊ _step‚238 ' lr-AdamW/pg1‚0.00019193208364065518 ' lr-AdamW/pg2‚0.00019193208364065518 trainer/global_step‚2949 $ _timestamp‚1.788430980757914e+09 _runtime‚2959.112950844]©]éÍËÇ $ _timestamp‚1.788430980757914e+09 _runtime‚2959.112950844 _step‚238 ' lr-AdamW/pg1‚0.00019193208364065518 ' lr-AdamW/pg2‚0.00019193208364065518 trainer/global_step‚2949îÕŸ=2Ì- _wandbruntime‚2959  _runtime‚2959ç[Ôè¸ÍøO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392465328749568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8747328456704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6282437628653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8747383471616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63378383 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52659865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31468040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _runtime‚2959.113911027 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405533904905728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6296175568384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349902811628544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398750869847141 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6293070175232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 % _timestamp‚1.7884309807586365e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6285541097573 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398747740950253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405530773121024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6296175568384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405530773121024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54565892 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23097855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341155428156928 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23098406 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341155400022528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚239 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116061353 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405533904905728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392465303321600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63401488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31468037 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52659859 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878X¼(ÎýO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _runtime‚2959.113911027 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405533904905728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6296175568384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349902811628544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398750869847141 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6293070175232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 % _timestamp‚1.7884309807586365e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6285541097573 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398747740950253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405530773121024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6296175568384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405530773121024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54565892 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23097855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341155428156928 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23098406 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341155400022528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚239 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116061353 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405533904905728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392465303321600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63401488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31468037 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52659859 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392465328749568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8747328456704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6282437628653 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8747383471616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63378383 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52659865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31468040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ïÑlݦ2Ï- _wandbruntime‚2960  _runtime‚2960Ÿ# ×'ЂN T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405667511321088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31478290 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398879246744569 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392594700322816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116076616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52677143 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63422604 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52677143 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63422604 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6287649890881 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23105443 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63399473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398882375641665 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116099753 % _timestamp‚1.7884309816483326e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6298287895552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405667511321088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚399369197033984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116099753 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚399369223425536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6295182489088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54583733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23105979 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31478293 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350018157533184 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8750263510528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405664379523072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52677149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54584272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6298287895552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350018074397184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6284546421753 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116076616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392594725750784º½läZ#чN MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23105443 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63399473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398882375641665 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116099753 % _timestamp‚1.7884309816483326e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6298287895552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405667511321088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚399369197033984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116099753 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚399369223425536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6295182489088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54583733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23105979 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31478293 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350018157533184 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8750263510528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405664379523072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52677149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54584272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6298287895552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350018074397184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6284546421753 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116076616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392594725750784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405667511321088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31478290 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398879246744569 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392594700322816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116076616 J=DeviceStatsMonitor.on_train_batch_eœ¬Ó³nd/active.large_pool.freed‚52677143 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63422604 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52677143 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63422604 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6287649890881ð.D.@2Ò- _wandbruntime‚2960  _runtime‚2960èWÛÄTÓÎ 1loss/SRqqt2/assignment_loss‚2.250813663557724 .loss/FRt4/detection_loss‚2.898071521712888 1loss/SRqqt3/detection_loss‚1.1986844141693676 0loss/SRqqt4/detection_loss‚2.771452377621944 /loss/FBt2/assignment_loss‚0.484619489102397 2loss/SRqqt3/assignment_loss‚1.4426078951753238 .loss/FRt2/detection_loss‚2.666707237561544 % _timestamp‚1.7884309816499734e+09 _runtime‚2960.004948741 /loss/FRt1/assignment_loss‚2.313800409220267 /loss/FRt4/assignment_loss‚5.340723006956039 1loss/SRqqt1/assignment_loss‚1.183396660188834 2loss/SRqqt4/assignment_loss‚1.9550573390263777 1loss/FBt4/assignment_loss‚0.40665971422020125 1loss/SRqqt1/detection_loss‚1.2908131651083627 -loss/FBt2/detection_loss‚2.85122150182724 &loss/total_loss‚48.335261159074065 trainer/global_step‚2949 _step‚241 .loss/FRt1/detection_loss‚1.639563891766728 /loss/FRt3/assignment_loss‚2.030247950553894 /loss/FRt3/detection_loss‚1.9707326085865498 .loss/FBt3/detection_loss‚2.251336404432853 /loss/FBt4/detection_loss‚1.4826336666969036 0loss/FBt1/assignment_loss‚0.1836595356440788 1loss/FBt3/assignment_loss‚0.41068699459234875 /loss/FBt1/detection_loss‚1.0541511714220682 /loss/FRt2/assignment_loss‚5.532520294189453 1loss/SRqqt2/detection_loss‚2.7251002457406788 epoch‚2_e=YÔÓ 1loss/FBt4/assignment_loss‚0.40665971422020125 1loss/SRqqt1/detection_loss‚1.2908131651083627 -loss/FBt2/detection_loss‚2.85122150182724 &loss/total_loss‚48.335261159074065 trainer/global_step‚2949 _step‚241 .loss/FRt1/detection_loss‚1.639563891766728 /loss/FRt3/assignment_loss‚2.030247950553894 /loss/FRt3/detection_loss‚1.9707326085865498 .loss/FBt3/detection_loss‚2.251336404432853 /loss/FBt4/detection_loss‚1.4826336666969036 0loss/FBt1/assignment_loss‚0.1836595356440788 1loss/FBt3/assignment_loss‚0.41068699459234875 /loss/FBt1/detection_loss‚1.0541511714220682 /loss/FRt2/assignment_loss‚5.532520294189453 1loss/SRqqt2/detection_loss‚2.7251002457406788 epoch‚2 1loss/SRqqt2/assignment_loss‚2.250813663557724 .loss/FRt4/detection_loss‚2.898071521712888 1loss/SRqqt3/detection_loss‚1.1986844141693676 0loss/SRqqt4/detection_loss‚2.771452377621944 /loss/FBt2/assignment_loss‚0.484619489102397 2loss/SRqqt3/assignment_loss‚1.4426078951753238 .loss/FRt2/detection_loss‚2.666707237561544 % _timestamp‚1.7884309816499734e+09 _runtime‚2960.004948741 /loss/FRt1/assignment_loss‚2.313800409220267 /loss/FRt4/assignment_loss‚5.340723006956039 1loss/SRqqt1/assignment_loss‚1.183396660188834 2loss/SRqqt4/assignment_loss‚1.9550573390263777ñ¿»œÕ‚(:‘ •åÔ£À«µ gpu.0.gpu‚68.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚259.176' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.83519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Y«ç^Ö‚(:Ó •åÔܲõ¶ disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 108.4921875* disk./.usagePercent‚18.416608300362295 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 108.4921875# proc.memory.rssMB‚ 3040.46484375 network.sent‚ 6.528184e+06 disk.nvme0n1.in‚ 162.18359375% disk.nvme0n1.out‚22632.8681640625 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 162.18359375" disk.dm-0.out‚22632.8681640625 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda.in‚0$ memory_percent‚92.00372695922852 cpu‚25.24508141779691 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0' proc.memory.availableMB‚ 1310.109375 network.recv‚ 1.05553e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚22632.8681640625 disk.sda3.in‚0 disk.drbd1716.in‚0) proc.memory.percent‚18.55752468109131% disk./.usageGB‚274.42077255249023 disk.dm-1.out‚ 108.4921875”ɜׂ(:‘ ž•åÔÞÌÛ´ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚135.56' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.112000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó „›bØ‚(:× ž•åÔÓé¼· disk.dm-1.out‚ 108.83203125" disk.nvme0c0n1.in‚ 162.18359375 disk.drbd1716.out‚0$ disk./.usageGB‚274.4432907104492 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 108.83203125$ memory_percent‚92.00377464294434 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 3040.46484375 cpu‚25.244099971478093 disk.sda2.out‚0" disk.dm-0.out‚22676.2548828125 disk.sda3.out‚ 108.83203125( proc.memory.availableMB‚ 1310.1015625) proc.memory.percent‚18.55752468109131 network.sent‚ 6.63677e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 162.18359375 disk.drbd1716.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.418119512836913 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125' disk.nvme0c0n1.out‚22676.2548828125 disk.sda.in‚0% disk.nvme0n1.out‚22676.2548828125 disk.drbd2080.out‚0 network.recv‚ 1.061392e+06 disk.sda2.in‚0›£,Ù‚(:’ ­•åÔ´Àø´ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚301.509' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.30180000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ösì0HÚ‚(:½ ­•åÔÛ›þ· disk.sda3.out‚ 109.65625 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1309.81640625 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375 disk.dm-0.out‚ 22721.1875 disk.sda1.in‚0 disk.drbd2265.out‚0 cpu‚25.243211459327046 disk.nvme0n1.out‚ 22721.1875 disk.sda2.in‚0 disk.sda.out‚ 109.65625 disk.rbd0.out‚0$ memory_percent‚92.00551509857178 network.sent‚ 6.642946e+06) disk./.usagePercent‚18.41843261005129! disk.nvme0c0n1.out‚ 22721.1875 disk.sda3.in‚0 network.recv‚ 1.066185e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 3040.46484375) proc.memory.percent‚18.55752468109131 proc.cpu.threads‚17$ disk./.usageGB‚274.4479560852051 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 109.65625 disk.rbd1.in‚0}hÃ!2Û- _wandbruntime‚3006  _runtime‚3006×d]ÉÜà _runtime‚3006.163353117 _step‚242 ' lr-AdamW/pg1‚0.00019121176667010045 ' lr-AdamW/pg2‚0.00019121176667010045 trainer/global_step‚2999 % _timestamp‚1.7884310278083403e+09y~ÎÝÈ trainer/global_step‚2999 % _timestamp‚1.7884310278083403e+09 _runtime‚3006.163353117 _step‚242 ' lr-AdamW/pg1‚0.00019121176667010045 ' lr-AdamW/pg2‚0.00019121176667010045òáñhþ2Þ- _wandbruntime‚3006  _runtime‚3006 9n‰ý'ß÷O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412211107637248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398935153382400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6387890874861 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355670294135296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412214239421952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64436235 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8894322771968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚405812407355392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6398700281856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 _runtime‚3006.164380897 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412214239421952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64459340 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405812433746944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚243 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6390994343781 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6398700281856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8894265659904 $ _timestamp‚1.788431027809081e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55459747 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346775971363328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405323044257261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53524059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23479057 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53524065 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚405812407355392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401805675008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53524065 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53524059 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23478498 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405326173154149 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401805675008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚2999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117960294 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412211107637248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117983405 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31980687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚405812433746944æ&l(àüO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398935153382400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6387890874861 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355670294135296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412214239421952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64436235 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8894322771968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚405812407355392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6398700281856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 _runtime‚3006.164380897 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412214239421952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64459340 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405812433746944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚243 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6390994343781 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6398700281856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8894265659904 $ _timestamp‚1.788431027809081e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55459747 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346775971363328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405323044257261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53524059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23479057 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53524065 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚405812407355392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401805675008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53524065 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53524059 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23478498 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405326173154149 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401805675008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚2999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117960294 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412211107637248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117983405 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31980687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚405812433746944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412211107637248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55477017 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 trainer/global_step‚2999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6403918002176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412344714039296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6400812595712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚346888354093056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53541349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405943901443584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405943901443584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31990943 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2508644403OÀ7ù!2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64457325 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355785560664576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405457678948673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64457325 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚405943927835136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118021805 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399064575811584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _step‚244 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64480456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405943927835136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23486609 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31990940 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405454550051577 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6400812595712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6403918002176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117998668 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 % _timestamp‚1.7884310287048445e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚355785645897728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399064550383616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6389999667961 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8897263670272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412347845837312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117998668 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64480456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412347845837312 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55477017 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 trainer/global_step‚2999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6403918002176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412344714039296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6400812595712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚346888354093056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53541349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405943901443584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405943901443584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31990943 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64457325 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355785560664576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405457678948673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64457325 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚405943927835136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118021805 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399064575811584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _step‚244 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64480456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405943927835136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23486609 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31990940 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405454550051577 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6400812595712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6403918002176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117998668 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 % _timestamp‚1.7884310287048445e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚355785645897728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399064550383616ôŽ­Ÿú2ä- _wandbruntime‚3007  _runtime‚3007búwUåÏ /loss/FRt1/detection_loss‚2.1221152828740224 /loss/FRt3/detection_loss‚2.0688309904004707 0loss/SRqqt2/detection_loss‚2.783475174948021 /loss/SRqqt3/detection_loss‚1.23695439853695 /loss/FBt4/detection_loss‚1.5802976070293064 /loss/FRt1/assignment_loss‚2.359458207019738 1loss/FBt2/assignment_loss‚0.47124365584638256 0loss/SRqqt4/detection_loss‚2.646325843334198 epoch‚2 2loss/SRqqt3/assignment_loss‚1.4043661015254727 % _timestamp‚1.7884310287066998e+09 2loss/SRqqt1/assignment_loss‚1.2623936295801519 2loss/SRqqt2/assignment_loss‚1.8980678766965866 1loss/FBt1/assignment_loss‚0.15061421899865707 0loss/FBt4/assignment_loss‚0.2643144617440748 0loss/SRqqt1/detection_loss‚1.739695298321107 /loss/FBt1/detection_loss‚1.0544815685881197 .loss/FBt2/detection_loss‚3.131376096835503 /loss/FRt3/assignment_loss‚2.402558678486308 .loss/FBt3/detection_loss‚2.357417117465626 _runtime‚3007.061656591 /loss/FRt4/detection_loss‚3.0144048664304943 /loss/FRt2/assignment_loss‚4.852825324875968 1loss/SRqqt4/assignment_loss‚2.051174973845482 _step‚245 %loss/total_loss‚49.31121202350632 trainer/global_step‚2999 /loss/FRt2/detection_loss‚2.7163441487721034 /loss/FRt4/assignment_loss‚5.394919819302029 0loss/FBt3/assignment_loss‚0.34755668204955081l`ÔZæÔ /loss/FRt3/detection_loss‚2.0688309904004707 0loss/SRqqt2/detection_loss‚2.783475174948021 /loss/SRqqt3/detection_loss‚1.23695439853695 /loss/FBt4/detection_loss‚1.5802976070293064 /loss/FRt1/assignment_loss‚2.359458207019738 1loss/FBt2/assignment_loss‚0.47124365584638256 0loss/SRqqt4/detection_loss‚2.646325843334198 epoch‚2 2loss/SRqqt3/assignment_loss‚1.4043661015254727 % _timestamp‚1.7884310287066998e+09 2loss/SRqqt1/assignment_loss‚1.2623936295801519 2loss/SRqqt2/assignment_loss‚1.8980678766965866 1loss/FBt1/assignment_loss‚0.15061421899865707 0loss/FBt4/assignment_loss‚0.2643144617440748 0loss/SRqqt1/detection_loss‚1.739695298321107 /loss/FBt1/detection_loss‚1.0544815685881197 .loss/FBt2/detection_loss‚3.131376096835503 /loss/FRt3/assignment_loss‚2.402558678486308 .loss/FBt3/detection_loss‚2.357417117465626 _runtime‚3007.061656591 /loss/FRt4/detection_loss‚3.0144048664304943 /loss/FRt2/assignment_loss‚4.852825324875968 1loss/SRqqt4/assignment_loss‚2.051174973845482 _step‚245 %loss/total_loss‚49.31121202350632 trainer/global_step‚2999 /loss/FRt2/detection_loss‚2.7163441487721034 /loss/FRt4/assignment_loss‚5.394919819302029 0loss/FBt3/assignment_loss‚0.3475566820495508 /loss/FRt1/detection_loss‚2.1221152828740224õ)åø“ç‚(:ˆ ¼•åÔÜ俶 gpu.0.gpu‚65.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚302.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.4904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ý¤¿Nè‚(:à ¼•åÔÿåɸ network.sent‚ 6.648212e+06 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 cpu‚25.242247954147505$ disk./.usageGB‚274.4652557373047 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 162.18359375 disk.dm-0.out‚ 22768.5859375 proc.memory.rssMB‚ 3040.46875) disk./.usagePercent‚18.41959360421907 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 162.18359375 network.recv‚ 1.069392e+06 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 109.921875 disk.rbd1.out‚ 1.533203125" disk.nvme0n1.out‚ 22768.5859375 disk.drbd2266.in‚0 disk.sda3.out‚ 109.921875$ memory_percent‚92.00420379638672 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 1310.03125 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 22768.5859375) proc.memory.percent‚18.55754852294922 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 109.921875Ï°O¡‘é‚(:† Ë•åÔ—›×µ gpu.0.gpu‚47.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚129.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à4E4aê‚(:Ö Ë•åÔÄ•ç¶ disk.drbd1716.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚22842.0126953125 disk.drbd2266.in‚0 network.recv‚ 1.075545e+06* disk./.usagePercent‚18.420955820308762 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375" disk.dm-0.out‚22842.0126953125 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 162.18359375$ memory_percent‚92.00575351715088 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚274.4855537414551 disk.sda3.out‚ 110.62890625 disk.drbd2080.out‚0 disk.rbd1.out‚ 1.533203125' disk.nvme0c0n1.out‚22842.0126953125 proc.memory.rssMB‚ 3040.46875 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚25.241252906769546 disk.sda1.out‚0 network.sent‚ 6.756875e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 110.62890625 disk.drbd2265.in‚0 disk.dm-1.out‚ 110.62890625) proc.memory.availableMB‚ 1309.77734375) proc.memory.percent‚18.55754852294922þȱW“ë‚(:ˆ Ú•åÔš‡‘µ gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.788' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.5576 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽøÆaì‚(:Ö Ú•åÔ‘㥶 disk.sda.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1309.77734375 proc.memory.rssMB‚ 3040.46875 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚22887.6884765625 disk.dm-0.in‚ 162.18359375% disk./.usageGB‚274.45255279541016 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 111.23828125" disk.nvme0c0n1.in‚ 162.18359375' disk.nvme0c0n1.out‚22887.6884765625) proc.memory.percent‚18.55754852294922 disk.sda.out‚ 111.23828125 disk.rbd1.in‚0" disk.dm-0.out‚22887.6884765625 cpu‚25.240246126307937 disk.drbd1716.out‚0 disk.drbd2265.in‚0 network.sent‚ 6.762663e+06 network.recv‚ 1.079536e+06) disk./.usagePercent‚18.41874109913005 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 111.23828125 disk.rbd1.out‚ 1.533203125$ memory_percent‚92.00575351715088 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0£³ôm2í- _wandbruntime‚3053  _runtime‚3053„"wÈî ' lr-AdamW/pg1‚0.00019047291753910995 ' lr-AdamW/pg2‚0.00019047291753910995 trainer/global_step‚3049 $ _timestamp‚1.788431074955331e+09 _runtime‚3053.310438465 _step‚246—|²¿ÍïÇ _runtime‚3053.310438465 _step‚246 ' lr-AdamW/pg1‚0.00019047291753910995 ' lr-AdamW/pg2‚0.00019047291753910995 trainer/global_step‚3049 $ _timestamp‚1.788431074955331e+09ö\›‹2ð- _wandbruntime‚3053  _runtime‚3053¶5¡ñÁñøO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65517192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚418894573938176 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884310749560983e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚418891442153472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411901476461157 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6507435781632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418891442153472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405405028871168 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041171438080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6493344121069 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352396486435328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412387111764992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65494087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361437602858496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚411898347564269 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6496447589989 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6504330388480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6504330388480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32493337 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412387138156544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9041116423168 trainer/global_step‚3049 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54388265 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105Ñ7«¼(òýO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54388259 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 _runtime‚3053.311383368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6504330388480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6504330388480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32493337 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412387138156544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9041116423168 trainer/global_step‚3049 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54388265 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65517192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚418894573938176 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884310749560983e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚418891442153472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411901476461157 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6507435781632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418891442153472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405405028871168 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041171438080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6493344121069 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352396486435328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412387111764992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65494087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361437602858496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚411898347564269 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6496447589989 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54405543 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119943857 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32503593 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419028180353536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419025048555520 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884310758444903e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54405549 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56373140 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119920720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412518605853184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412029853358585 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6509548108800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6509548108800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32503590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412032982255681 _step‚248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56373667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361552953555456 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6506442702336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚412518605853184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65538308 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23869550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720•ôÍ 'õ‡N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412518605853184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412029853358585 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6509548108800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6509548108800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32503590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412032982255681 _step‚248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56373667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361552953555456 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6506442702336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚412518605853184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65538308 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23869550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65538308 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361553036691456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9044056256000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405534425872384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119943857 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419028180353536 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6506442702336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119920720 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54405543 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119943857 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32503593 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419028180353536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419025048555520 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884310758444903e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54405549 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56373140 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119920720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ø„Fa2ö- _wandbruntime‚3054  _runtime‚3054‘À¦÷Ö /loss/FRt1/detection_loss‚1.9961066677102022 1loss/SRqqt1/detection_loss‚1.3069420382380486 trainer/global_step‚3049 /loss/FRt2/detection_loss‚3.0042106178071766 % _timestamp‚1.7884310758462265e+09 /loss/FRt4/detection_loss‚3.1628220081329346 1l¦ë]\Foss/SRqqt3/detection_loss‚1.1777247199452068 2loss/SRqqt1/assignment_loss‚1.4215918148439082 2loss/SRqqt2/assignment_loss‚1.6453651679823031 1loss/FBt3/assignment_loss‚0.04880596562806103 .loss/FBt2/detection_loss‚2.978700260321299 _runtime‚ 3054.2011569 /loss/FRt4/assignment_loss‚5.214397925596971 2loss/SRqqt3/assignment_loss‚1.3321917213679078 1loss/FBt2/assignment_loss‚0.38915871580437056 -loss/FRt3/detection_loss‚2.00090768347021 /loss/FBt1/detection_loss‚1.3401089583337307 0loss/FRt2/assignment_loss‚5.0120077398088245 1loss/FBt4/assignment_loss‚0.25127830317796906 0loss/SRqqt2/detection_loss‚2.748423383786128 epoch‚2 _step‚249 0loss/FRt1/assignment_loss‚2.8073726828609193 1loss/FBt1/assignment_loss‚0.13671039102868618 1loss/SRqqt4/detection_loss‚2.6672514033944985 /loss/FBt4/detection_loss‚1.4097126417327672 &loss/total_loss‚50.369869380273585 2loss/SRqqt4/assignment_loss‚2.7802319432559766 /loss/FBt3/detection_loss‚2.8268529110484653 /loss/FRt3/assignment_loss‚2.710993714997026„}µaøÛ 2loss/SRqqt1/assignment_loss‚1.4215918148439082 2loss/SRqqt2/assignment_loss‚1.6453651679823031 1loss/FBt3/assignment_loss‚0.04880596562806103 .loss/FBt2/detection_loss‚2.978700260321299 _runtime‚ 3054.2011569 /loss/FRt4/assignment_loss‚5.214397925596971 2loss/SRqqt3/assignment_loss‚1.3321917213679078 1loss/FBt2/assignment_loss‚0.38915871580437056 -loss/FRt3/detection_loss‚2.00090768347021 /loss/FBt1/detection_loss‚1.3401089583337307 0loss/FRt2/assignment_loss‚5.0120077398088245 1loss/FBt4/assignment_loss‚0.25127830317796906 0loss/SRqqt2/detection_loss‚2.748423383786128 epoch‚2 _step‚249 0loss/FRt1/assignment_loss‚2.8073726828609193 1loss/FBt1/assignment_loss‚0.13671039102868618 1loss/SRqqt4/detection_loss‚2.6672514033944985 /loss/FBt4/detection_loss‚1.4097126417327672 &loss/total_loss‚50.369869380273585 2loss/SRqqt4/assignment_loss‚2.7802319432559766 /loss/FBt3/detection_loss‚2.8268529110484653 /loss/FRt3/assignment_loss‚2.710993714997026 /loss/FRt1/detection_loss‚1.9961066677102022 1loss/SRqqt1/detection_loss‚1.3069420382380486 trainer/global_step‚3049 /loss/FRt2/detection_loss‚3.0042106178071766 % _timestamp‚1.7884310758462265e+09 /loss/FRt4/detection_loss‚3.1628220081329346 1loss/SRqqt3/detection_loss‚1.1777247199452068ùÁ½ٞù‚(:“ é•åÔ„»’µ gpu.0.gpu‚87.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚244.063' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.812599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z¦Á\ú‚(:Ñ é•åÔÜýÑ· disk.drbd2080.in‚0 disk.sda.out‚ 111.765625) proc.memory.percent‚18.55754852294922 disk.sda3.out‚ 111.765625 disk.drbd2265.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 3040.46875 network.recv‚ 1.084312e+06 disk.nvme0n1.in‚ 162.18359375 disk.drbd1716.in‚0# disk.dm-0.out‚22932.79248046875 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 111.765625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2265.out‚0$ memory_percent‚92.00773239135742 cpu‚25.23937949005094% disk./.usageGB‚274.47644424438477 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1309.453125 proc.cpu.threads‚17 network.sent‚ 6.768938e+06 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 162.18359375( disk.nvme0c0n1.out‚22932.79248046875 disk.drbd2080.out‚0* disk./.usagePercent‚18.420344474316998& disk.nvme0n1.out‚22932.79248046875 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 162.18359375à´JÍžû‚(:“ ø•åÔªÃØ´ gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚136.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.276400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EOdü‚(:Ù ø•åԇ𳷠disk.nvme0n1.in‚ 162.18359375) proc.memory.availableMB‚ 1309.70703125* disk./.usagePercent‚18.420853673302602 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda3.in‚0 disk.drbd2265.in‚0$ memory_percent‚92.00618267059326 proc.cpu.threads‚17 disk.rbd1.in‚0 network.sent‚ 6.877088e+06 network.recv‚ 1.089831e+06 disk.drbd2080.in‚0 disk.sda.out‚ 112.22265625$ disk./.usageGB‚274.4840316772461 disk.drbd2266.in‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚22974.81884765625 disk.sda3.out‚ 112.22265625 disk.rbd1.out‚ 1.533203125 disk.sda.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 112.22265625# disk.dm-0.out‚22974.81884765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 3040.46875 disk.dm-0.in‚ 162.18359375 disk.drbd1716.in‚0& disk.nvme0n1.out‚22974.81884765625 disk.sda2.out‚0) proc.memory.percent‚18.55754852294922 cpu‚25.238386785407645 disk.drbd1716.out‚0 disk.drbd2265.out‚0çµwדý‚(:ˆ ‡–åÔÑâ’µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚284.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÜjJWþ‚(:Ì ‡–åÔ±œñ¶& disk.nvme0n1.out‚23032.93603515625 disk.sda2.in‚0% disk./.usageGB‚274.50707626342773" disk.nvme0c0n1.in‚ 162.18359375 disk.sda1.in‚0 cpu‚25.237408166527175 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 3040.46875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.dm-1.out‚113.1875 disk.sda3.in‚0 disk.sda3.out‚113.1875 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 162.18359375) proc.memory.percent‚18.55754852294922* disk./.usagePercent‚18.422400214816946 disk.rbd0.in‚0$ memory_percent‚92.00620651245117 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.619140625' proc.memory.availableMB‚ 1309.703125 proc.cpu.threads‚17 network.recv‚ 1.094165e+06 disk.drbd2265.in‚0 network.sent‚ 6.883029e+06( disk.nvme0c0n1.out‚23032.93603515625 disk.dm-0.in‚ 162.18359375# disk.dm-0.out‚23032.93603515625 disk.sda.out‚113.1875 disk.rbd0.out‚0^V”à2ÿ- _wandbruntime‚3100  _runtime‚3100Ü Œ‡É€Ã _step‚250 ' lr-AdamW/pg1‚0.00018971569022273516 ' lr-AdamW/pg2‚0.00018971569022273516 trainer/global_step‚3099 % _timestamp‚1.7884311221190932e+09 _runtime‚3100.474131102‚‘öfÎÈ ' lr-AdamW/pg1‚0.00018971569022273516 ' lr-AdamW/pg2‚0.00018971569022273516 trainer/global_step‚3099 % _timestamp‚1.7884311221190932e+09 _runtime‚3100.474131102 _step‚250úñW´Þ2‚- _wandbruntime‚3100  _runtime‚3100œ,”;þ'ƒøO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55252459 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6601900836197 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6609960495104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358017057776128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55252465 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57249868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66551939 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6613065888256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367204672303616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418473650871277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66575044 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6598797367277 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425574908454400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24243881 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33005987 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367204589154304 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418476779768165 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425571776669696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425574908454400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57250411 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884311221198707e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411874853504000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3100.475145627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418961816174592 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9187559512576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6613065888256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425571776669696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚3099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418961842566144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121804398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66551939 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418961842566144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55252465 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358017029641728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9187614527488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 Nz(„ýO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121804398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66551939 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418961842566144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55252465 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358017029641728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9187614527488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55252459 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6601900836197 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6609960495104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358017057776128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55252465 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57249868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66551939 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6613065888256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367204672303616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418473650871277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66575044 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6598797367277 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425574908454400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24243881 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33005987 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367204589154304 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3275873280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418476779768165 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425571776669696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425574908454400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57250411 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884311221198707e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411874853504000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3100.475145627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418961816174592 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9187559512576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6613065888256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425571776669696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚3099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418961842566144ûÚ¾Ö2…- _wandbruntime‚3101  _runtime‚3101ÜÕ‹ïÕ†N BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121865909 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367320005559296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358129440505856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6604009629505 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419093336654336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419093336654336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418608285562689 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615178215424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6612072808960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121842772 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6600906160377 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55269749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57267847 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358129468640256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55269743 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24251607 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425705383071744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121842772 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚252ýg©Ì '‡†N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418608285562689 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615178215424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6612072808960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121842772 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6600906160377 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55269749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57267847 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358129468640256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55269743 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24251607 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425705383071744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121842772 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚252 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121865909 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367320005559296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83136000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358129440505856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6604009629505 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419093336654336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419093336654336üXiò¨2ˆ- _wandbruntime‚3101  _runtime‚3101$îüW‰Ñ 0loss/SRqqt2/detection_loss‚2.892298909334036 &loss/total_loss‚49.734728676205364 $ _timestamp‚1.788431123014622e+09 2loss/SRqqt4/assignment_loss‚2.0682407622536023 1loss/SRqqt2/assignment_loss‚2.329338158552463 /loss/FBt1/assignment_loss‚0.228820001436132 1loss/SRqqt1/detection_loss‚1.3332900253957825 _step‚253 0loss/FBt2/assignment_loss‚0.2914161355389903 trainer/global_step‚3099 _runtime‚3101.369595182 2loss/SRqqt1/assignment_loss‚1.3067283960439167 /loss/FBt4/assignment_loss‚0.475874253064012 .loss/FBt3/detection_loss‚2.102197392628743 /loss/FBt4/detection_loss‚1.9309312403202057 /loss/FRt4/assignment_loss‚4.868622121810913 1loss/SRqqt3/assignment_loss‚1.582640739007928 1loss/FBt3/assignment_loss‚0.21226752437024304 /loss/FRt1/detection_loss‚1.7367790271838506 /loss/FRt4/detection_loss‚3.0959218406677245 1loss/SRqqt3/detection_loss‚1.1788866749011921 /loss/FBt2/detection_loss‚3.3147230479452343 0loss/FRt1/assignment_loss‚2.0639300123850504 0loss/FRt3/assignment_loss‚3.1229631983000656 .loss/FRt3/detection_loss‚2.110671161320703 1loss/SRqqt4/detection_loss‚2.4288404405117037 /loss/FRt2/assignment_loss‚5.774434204101563 /loss/FRt2/detection_loss‚2.5884008741378786 /loss/FBt1/detection_loss‚0.6965125349934324 epoch‚2Jã9\ŠÖ /loss/FRt2/detection_loss‚2.5884008741378786 /loss/FBt1/detection_loss‚0.6965125349934324 epoch‚2 0loss/SRqqt2/detection_loss‚2.892298909334036 &loss/total_loss‚49.734728676205364 $ _timestamp‚1.788431123014622e+09 2loss/SRqqt4/assignment_loss‚2.0682407622536023 1loss/SRqqt2/assignment_loss‚2.329338158552463 /loss/FBt1/assignment_loss‚0.228820001436132 1loss/SRqqt1/detection_loss‚1.3332900253957825 _step‚253 0loss/FBt2/assignment_loss‚0.2914161355389903 trainer/global_step‚3099 _runtime‚3101.369595182 2loss/SRqqt1/assignment_loss‚1.3067283960439167 /loss/FBt4/assignment_loss‚0.475874253064012 .loss/FBt3/detection_loss‚2.102197392628743 /loss/FBt4/detection_loss‚1.9309312403202057 /loss/FRt4/assignment_loss‚4.868622121810913 1loss/SRqqt3/assignment_loss‚1.582640739007928 1loss/FBt3/assignment_loss‚0.21226752437024304 /loss/FRt1/detection_loss‚1.7367790271838506 /loss/FRt4/detection_loss‚3.0959218406677245 1loss/SRqqt3/detection_loss‚1.1788866749011921 /loss/FBt2/detection_loss‚3.3147230479452343 0loss/FRt1/assignment_loss‚2.0639300123850504 0loss/FRt3/assignment_loss‚3.1229631983000656 .loss/FRt3/detection_loss‚2.110671161320703 1loss/SRqqt4/detection_loss‚2.4288404405117037 /loss/FRt2/assignment_loss‚5.774434204101563ýœ*nZ‹‚(:Ï ––åÔÈñæ¶ network.recv‚ 1.098309e+06 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 113.765625 disk.rbd1.in‚0 disk.sda3.out‚ 113.765625 disk.sda1.in‚0 cpu‚25.23651168356461 network.sent‚ 6.888912e+06* disk./.usagePercent‚18.422797538510086 disk.sda.in‚0 disk.rbd1.out‚ 1.681640625# disk.dm-0.out‚23072.70556640625 disk.sda.out‚ 113.765625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda2.in‚0) proc.memory.percent‚18.55754852294922# disk./.usageGB‚274.512996673584 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0' proc.memory.availableMB‚ 1309.671875" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375 disk.sda1.out‚0 proc.memory.rssMB‚ 3040.46875 disk.drbd2080.out‚0 disk.drbd2265.out‚0( disk.nvme0c0n1.out‚23072.70556640625 disk.drbd1716.out‚0$ memory_percent‚92.00639724731445 disk.rbd0.in‚0& disk.nvme0n1.out‚23072.70556640625 disk.sda3.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17À+Œ‚(:’ ––åÔ®´ê¸ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚259.49' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.897999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^{‘‚(:† ¥–åÔôÄû´ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚278.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Ö[Ž‚(:Ð ¥–åÔÞÑ· disk.sda1.in‚0 disk.dm-1.out‚ 114.546875 disk.rbd1.out‚ 1.681640625 disk.drbd1716.out‚0 disk.sda.out‚ 114.546875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23116.58544921875 disk.dm-0.in‚ 162.18359375 network.sent‚ 6.997927e+06 disk.drbd1716.in‚0 disk.drbd2080.out‚0 proc.memory.rssMB‚ 3040.46875 network.recv‚ 1.104996e+06$ disk./.usageGB‚274.5697822570801 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 162.18359375# disk.dm-0.out‚23116.58544921875 disk.sda3.in‚0 disk.sda3.out‚ 114.546875 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0) disk./.usagePercent‚18.42660846666479 disk.drbd2265.out‚0& disk.nvme0n1.out‚23116.58544921875 disk.drbd2266.out‚0$ memory_percent‚92.00949668884277( proc.memory.availableMB‚ 1309.1640625) proc.memory.percent‚18.55754852294922 cpu‚25.23568795160239 disk.dm-1.in‚0 proc.cpu.threads‚174—”“‚(:ˆ ´–åÔƒ²—· gpu.0.gpu‚84.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ÊX½Y‚(:Î ´–åÔå‚㺠disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.drbd2080.in‚0 disk.sda.out‚114.9375 disk.drbd2265.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚23156.24169921875 network.sent‚ 7.005379e+06( disk.nvme0c0n1.out‚23156.24169921875 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1309.91015625 network.recv‚ 1.108377e+06 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 162.18359375$ memory_percent‚92.00494289398193 proc.cpu.threads‚17* disk./.usagePercent‚18.423924483675812 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.7138671875 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚114.9375 proc.memory.rssMB‚ 3040.46875) proc.memory.percent‚18.55754852294922 disk.dm-0.in‚ 162.18359375# disk.dm-0.out‚23156.24169921875 disk.sda3.out‚114.9375 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚25.23475420287042% disk./.usageGB‚274.52978897094727 disk.sda1.out‚0 disk.dm-1.in‚0w'2‘- _wandbruntime‚3147  _runtime‚3147ýf ÅÉ’Ã % _timestamp‚1.7884311694969559e+09 _runtime‚3147.851991028 _step‚254 ' lr-AdamW/pg1‚0.00018894024252601388 ' lr-AdamW/pg2‚0.00018894024252601388 trainer/global_step‚3149æäI#Î“È % _timestamp‚1.7884311694969559e+09 _runtime‚3147.851991028 _step‚254 ' lr-AdamW/pg1‚0.00018894024252601388 ' lr-AdamW/pg2‚0.00018894024252601388 trainer/global_step‚3149þë—ŒÚ2”- _wandbruntime‚3147  _runtime‚3147Û(²£$•÷O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432252111185920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24623484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58142121 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33518637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372971568010752 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56116659 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425052083075173 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425536546975744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67609791 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432252111185920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123749561 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67632896 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432255242970624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33518640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0™¸UaZ E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6715590601728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425536546975744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6707354082405 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6718695994880a)¤(–üO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33518637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372971568010752 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56116659 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425052083075173 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425536546975744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67609791 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432252111185920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123749561 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67632896 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432255242970624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33518640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6715590601728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425536546975744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6707354082405 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6718695994880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432252111185920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24623484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58142121 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302ÿ/5|2—- _wandbruntime‚3148  _runtime‚3148|òOð'˜‚N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425668041063936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159982 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58160519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24631626 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432388849385984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425668014672384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67654012 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418474100566016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3148.751431295 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33528893 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56133943 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67654012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67630881 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425668041063936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123787961 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6720808322048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363750011846656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123764824 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425668014672384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56133949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚432385717587968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123764824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123787961 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6709462875713 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6717702915584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24631092 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425183588869697 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373086817998336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363749983712256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425668014672384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56133949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚432385717587968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123764824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123787961 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6709462875713 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6717702915584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24631092 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425183588869697 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373086817998336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363749983712256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159982 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58160519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24631626 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432388849385984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425668014672384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67654012 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418474100566016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3148.751431295 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33528893 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56133943 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67654012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67630881 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425668041063936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123787961 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6720808322048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363750011846656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123764824 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56980859 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125648502 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56980865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 % _timestamp‚1.7884312166775165e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25006063 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480321242112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438932445702144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432111251385344 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34031290 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6821220708352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378738437296640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424814553625600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6824326101504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68667643 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6809703859693 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59037918 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480378354176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431624257485293 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68690748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6812807328613 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6821220708352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68690748 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34031290 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6821220708352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378738437296640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424814553625600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6824326101504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68667643 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6809703859693 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59037918 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480378354176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431624257485293 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68690748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6812807328613 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6821220708352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68690748 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56980859 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125648502 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56980865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 % _timestamp‚1.7884312166775165e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25006063 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480321242112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438932445702144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432111251385344 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125710013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚432242719081984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚432242719081984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25014172 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9483239740928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 VGDeviceStatsMonitor.on_train_batch_end/¬€ &reserved_bytes.small_pool.current‚ 3938451456 _step‚260 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68711864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68688733 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68711864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378853851892736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9483296839680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125710013 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25013614 $ _timestamp‚1.788431217579307e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34041543 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚369370555053056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439069183902208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125686876 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431755763279609 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56998143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚432242745473536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6826438428672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56998149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439066052104192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59055154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚3199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125686876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456ÔeŽ '­†N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125710013 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25013614 $ _timestamp‚1.788431217579307e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34041543 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚369370555053056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439069183902208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125686876 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431755763279609 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56998143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚432242745473536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6826438428672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56998149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439066052104192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59055154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚3199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125686876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6823333022208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125710013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚432242719081984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚432242719081984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25014172 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9483239740928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _step‚260 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68711864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68688733 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68711864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378853851892736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9483296839680„ê'¤õ2®- _wandbruntime‚3195  _runtime‚3195¤EèÿZ¯Ô /loss/FRt1/detection_loss‚1.9226909149438143 _step‚261 /loss/FRt2/assignment_loss‚4.463577434934419 2loss/SRqqt3/assignment_loss‚1.1736360143708147 1loss/SRqqt1/detection_loss‚1.2742070257663727 1loss/SRqqt4/assignment_loss‚2.132602406175513 .loss/FRt3/detection_loss‚2.150355478579348 /loss/FBt3/detection_loss‚2.1507023800821865 .loss/FBt4/detection_loss‚1.289804970694555 /loss/FRt4/detection_loss‚3.1174170351028443 % _timestamp‚1.7884312175809867e+09 0loss/FRt1/assignment_loss‚2.4518410935997963 1loss/FBt4/assignment_loss‚0.48971025363268406 1loss/SRqqt2/detection_loss‚2.8579155372248755 1loss/SRqqt3/detection_loss‚1.4597491377937621 1loss/SRqqt4/detection_loss‚2.8493439837505945 epoch‚2 trainer/global_step‚3199 /loss/FRt3/assignment_loss‚2.655052114616741 /loss/FRt4/assignment_loss‚4.717765102386474 0loss/FBt3/assignment_loss‚0.2721477819757317 /loss/FBt1/detection_loss‚1.2196161263407423 _runtime‚3195.935976249 2loss/SRqqt1/assignment_loss‚1.5893398061604567 0loss/SRqqt2/assignment_loss‚1.96012196275923 1loss/FBt2/assignment_loss‚0.30802756850607693 .loss/FRt2/detection_loss‚2.693077535464846 1loss/FBt1/assignment_loss‚0.03143441490380912 /loss/FBt2/detection_loss‚2.6740134278933207 &loss/total_loss‚47.904149507659014XÿD_°Ù % _timestamp‚1.7884312175809867e+09 0loss/FRt1/assignment_loss‚2.4518410935997963 1loss/FBt4/assignment_loss‚0.48971025363268406 1loss/SRqqt2/detection_loss‚2.8579155372248755 1loss/SRqqt3/detection_loss‚1.4597491377937621 1loss/SRqqt4/detection_loss‚2.8493439837505945 epoch‚2 trainer/global_step‚3199 /loss/FRt3/assignment_loss‚2.655052114616741 /loss/FRt4/assignment_loss‚4.717765102386474 0loss/FBt3/assignment_loss‚0.2721477819757317 /loss/FBt1/detection_loss‚1.2196161263407423 _runtime‚3195.935976249 2loss/SRqqt1/assignment_loss‚1.5893398061604567 0loss/SRqqt2/assignment_loss‚1.96012196275923 1loss/FBt2/assignment_loss‚0.30802756850607693 .loss/FRt2/detection_loss‚2.693077535464846 1loss/FBt1/assignment_loss‚0.03143441490380912 /loss/FBt2/detection_loss‚2.6740134278933207 &loss/total_loss‚47.904149507659014 /loss/FRt1/detection_loss‚1.9226909149438143 _step‚261 /loss/FRt2/assignment_loss‚4.463577434934419 2loss/SRqqt3/assignment_loss‚1.1736360143708147 1loss/SRqqt1/detection_loss‚1.2742070257663727 1loss/SRqqt4/assignment_loss‚2.132602406175513 .loss/FRt3/detection_loss‚2.150355478579348 /loss/FBt3/detection_loss‚2.1507023800821865 .loss/FBt4/detection_loss‚1.289804970694555 /loss/FRt4/detection_loss‚3.1174170351028443…žÿˆ“±‚(:ˆ ÿ–åÔΈ±µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.912' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{l°éY²‚(:Î ÿ–åÔùò‰·& disk.nvme0n1.out‚23372.17724609375 disk.sda3.in‚0 disk.sda3.out‚ 117.84375 disk.sda.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 117.84375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚92.00866222381592 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.in‚0 proc.memory.rssMB‚ 3040.46875% disk./.usageGB‚274.59988021850586 disk.dm-0.in‚ 162.18359375 disk.drbd2080.out‚0* disk./.usagePercent‚18.428628366109944# disk.dm-0.out‚23372.17724609375 disk.sda.out‚ 117.84375( disk.nvme0c0n1.out‚23372.17724609375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 network.sent‚ 7.135381e+06 network.recv‚ 1.13174e+06 disk.rbd1.out‚ 1.83984375" disk.nvme0c0n1.in‚ 162.18359375 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1309.30078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0) proc.memory.percent‚18.55754852294922 cpu‚25.23045480000235¼a±<ž³‚(:“ Ž—åÔ×Ö³µ gpu.0.gpu‚83.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.641' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.928200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜAv†`´‚(:Õ Ž—åÔض‰· disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 118.14453125 disk.drbd2265.in‚0& disk.nvme0n1.out‚23416.83154296875 disk.dm-0.in‚ 162.18359375# disk.dm-0.out‚23416.83154296875 disk.drbd1716.in‚0 proc.memory.rssMB‚ 3040.46875 disk.rbd1.out‚ 1.83984375 disk.sda3.in‚0 disk.drbd1716.out‚0 network.recv‚ 1.137676e+06 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 162.18359375( disk.nvme0c0n1.out‚23416.83154296875 disk.sda1.out‚0) proc.memory.percent‚18.55754852294922 cpu‚25.229545457957865 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚18.4294632066791% disk./.usageGB‚274.61231994628906 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ memory_percent‚92.01021194458008 network.sent‚ 7.244022e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0' proc.memory.availableMB‚ 1309.046875 disk.dm-1.out‚ 118.14453125 disk.sda3.out‚ 118.14453125ûD{’µ‚(:‡ —åÔ…Æœµ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚293.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.791 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Œì­f¶‚(:Û —åÔ¯ÖͶ network.recv‚ 1.142472e+06 disk.dm-0.in‚ 162.18359375 disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.18359375) proc.memory.availableMB‚ 1309.29296875$ disk./.usageGB‚274.6181335449219 disk.sda1.out‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚23465.03857421875 disk.rbd1.out‚ 1.83984375 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚23465.03857421875 disk.rbd0.in‚0 disk.drbd2266.out‚0$ memory_percent‚92.00870990753174 cpu‚25.228728204611595 network.sent‚ 7.250392e+06* disk./.usagePercent‚18.429853362161282 disk.drbd2080.out‚0 disk.sda.out‚ 118.83203125 disk.dm-1.in‚0 disk.dm-1.out‚ 118.83203125) proc.memory.percent‚18.55757236480713 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 162.18359375 disk.sda3.out‚ 118.83203125 disk.sda.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚23465.03857421875 disk.drbd1716.out‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 3040.47265625­iÅ2·- _wandbruntime‚3242  _runtime‚3242áÙ#‹É¸Ã trainer/global_step‚3249 % _timestamp‚1.7884312638868504e+09 _runtime‚3242.241903054 _step‚262 ' lr-AdamW/pg1‚0.00018733533616350508 ' lr-AdamW/pg2‚0.00018733533616350508Âû åÎ¹È ' lr-AdamW/pg1‚0.00018733533616350508 ' lr-AdamW/pg2‚0.00018733533616350508 trainer/global_step‚3249 % _timestamp‚1.7884312638868504e+09 _runtime‚3242.241903054 _step‚262†/‰d2º- _wandbruntime‚3242  _runtime‚3242Îg–#»øO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69725495 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6926850814976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9626838649856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445615912003072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59932805 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34543940 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384505526045184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445612780218368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34543937 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431284429114368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57845065 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57845065 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69748600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6929956208128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127593665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25388282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374878687395328 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6926850814976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384505438701568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445615912003072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59932219 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9626779440640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431284403686400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127570554 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 _runtime‚3242.242862595 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚438685955794944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 % _timestamp‚1.7884312638875418e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69725495 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438199560792301 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 % _timestamp‚1.7884312638875418e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69725495 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438199560792301 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445612780218368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34543937 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431284429114368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57845065 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57845065 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69748600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6929956208128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127593665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25388282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374878687395328 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6926850814976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384505438701568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445615912003072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59932219 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9626779440640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431284403686400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127570554 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 _runtime‚3242.242862595‡Wƒl2½- _wandbruntime‚3243  _runtime‚3243–S!>'¾ÿM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445749518418432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69746585 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69746585 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87330304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57862349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25396354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445749518418432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384620763841536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9629752912384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127608928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25395799 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚438334195483713 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6932068535296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57862343 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431413800687616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6917265899001 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438817423491584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57862349 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127632065 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59949989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438817449883136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57862343 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69769716 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6920369368129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚438331066586617 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 _runtime‚ 3243.14949357 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25396354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445749518418432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384620763841536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9629752912384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127608928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25395799 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚438334195483713 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6932068535296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57862343 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431413800687616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6917265899001 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438817423491584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57862349 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127632065 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59949989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438817449883136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57862343 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69769716 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6920369368129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚438331066586617 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 _runtime‚ 3243.14949357 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445749518418432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69746585 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69746585 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87330304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ˆwºo2À- _wandbruntime‚3243  _runtime‚3243p×úñÁÑ 0loss/FRt3/assignment_loss‚2.6195102977125266 /loss/FRt4/detection_loss‚3.1060076306263604 _step‚265 1loss/SRqqt4/assignment_loss‚2.040038116000317 /loss/FRt3/detection_loss‚2.4796187971767627 .loss/FBt2/detection_loss‚2.566367030143738 %loss/total_loss‚47.94522917214353 epoch‚2 % _timestamp‚1.7884312647956724e+09 .loss/FRt1/detection_loss‚1.909308897331357 2loss/SRqqt1/assignment_loss‚1.1162214782653432 1loss/SRqqt3/assignment_loss‚1.401818491394321 1loss/SRqqt4/detection_loss‚2.2218965115370573 1loss/SRqqt2/detection_loss‚2.6736645673002517 1loss/SRqqt3/detection_loss‚1.2938407432039578 2loss/SRqqt2/assignment_loss‚2.2909801444837026 0loss/FBt1/assignment_loss‚0.1410931319334456 /loss/FBt2/assignment_loss‚0.346385570338839 trainer/global_step‚3249 0loss/FRt1/assignment_loss‚2.6295792795717716 .loss/FBt3/detection_loss‚2.344439991495826 /loss/FBt1/detection_loss‚0.9741195908510217 /loss/FBt4/detection_loss‚1.4365203887449973 .loss/FRt2/detection_loss‚2.628067022737335 /loss/FRt2/assignment_loss‚5.150248036665075 /loss/FRt4/assignment_loss‚4.205913106600444 1loss/SRqqt1/detection_loss‚1.3135425245220012 0loss/FBt3/assignment_los¤~å fs‚0.7661073362029459 _runtime‚3243.150654281 1loss/FBt4/assignment_loss‚0.289940487304137967÷A\ÂÖ 1loss/FBt4/assignment_loss‚0.28994048730413796 0loss/FRt3/assignment_loss‚2.6195102977125266 /loss/FRt4/detection_loss‚3.1060076306263604 _step‚265 1loss/SRqqt4/assignment_loss‚2.040038116000317 /loss/FRt3/detection_loss‚2.4796187971767627 .loss/FBt2/detection_loss‚2.566367030143738 %loss/total_loss‚47.94522917214353 epoch‚2 % _timestamp‚1.7884312647956724e+09 .loss/FRt1/detection_loss‚1.909308897331357 2loss/SRqqt1/assignment_loss‚1.1162214782653432 1loss/SRqqt3/assignment_loss‚1.401818491394321 1loss/SRqqt4/detection_loss‚2.2218965115370573 1loss/SRqqt2/detection_loss‚2.6736645673002517 1loss/SRqqt3/detection_loss‚1.2938407432039578 2loss/SRqqt2/assignment_loss‚2.2909801444837026 0loss/FBt1/assignment_loss‚0.1410931319334456 /loss/FBt2/assignment_loss‚0.346385570338839 trainer/global_step‚3249 0loss/FRt1/assignment_loss‚2.6295792795717716 .loss/FBt3/detection_loss‚2.344439991495826 /loss/FBt1/detection_loss‚0.9741195908510217 /loss/FBt4/detection_loss‚1.4365203887449973 .loss/FRt2/detection_loss‚2.628067022737335 /loss/FRt2/assignment_loss‚5.150248036665075 /loss/FRt4/assignment_loss‚4.205913106600444 1loss/SRqqt1/detection_loss‚1.3135425245220012 0loss/FBt3/assignment_loss‚0.7661073362029459 _runtime‚3243.150654281‰“âÝQžÃ‚(:“ ¬—åÔŠ—¾µ gpu.0.gpu‚72.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.148599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ\‹~bÄ‚(:× ¬—åÔãåõ· disk.sda2.out‚0 network.recv‚ 1.146139e+06* disk./.usagePercent‚18.430233021334562( disk.nvme0c0n1.out‚23504.37451171875 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.18359375 disk.dm-1.in‚0 disk.sda3.out‚ 119.1328125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0) proc.memory.percent‚18.55757236480713$ disk./.usageGB‚274.6237907409668 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd2265.out‚0 cpu‚25.22789017976943 network.sent‚ 7.255999e+06 disk.sda2.in‚0 disk.dm-1.out‚ 119.1328125 disk.rbd1.out‚ 1.83984375 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0$ memory_percent‚92.00761318206787) proc.memory.availableMB‚ 1309.47265625" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375 disk.drbd2266.out‚0# disk.dm-0.out‚23504.37451171875 disk.drbd2080.out‚0 disk.sda.out‚ 119.1328125# proc.memory.rssMB‚ 3040.47265625 proc.cpu.threads‚17& disk.nvme0n1.out‚23504.37451171875©Ü@Å‚(:’ »—åÔ¦à•µ gpu.0.gpu‚60.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.304' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.86079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö€Ì]Æ‚(:Ò »—åÔ§ˆÅ¶ disk.rbd1.out‚ 1.83984375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚23549.35791015625 disk.sda3.out‚ 119.671875 cpu‚25.2271782554804 disk.drbd1716.in‚0 disk.sda2.in‚0 network.sent‚ 7.364472e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda.in‚0 disk.dm-0.in‚ 162.18359375 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚92.00921058654785 disk.dm-1.in‚0 disk.dm-1.out‚ 119.671875 network.recv‚ 1.151701e+06 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 119.671875 disk.drbd2080.in‚0) disk./.usagePercent‚18.43096110676194 disk.sda1.in‚0# proc.memory.rssMB‚ 3040.47265625) proc.memory.percent‚18.55757236480713% disk./.usageGB‚274.63463973999023 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0# disk.dm-0.out‚23549.35791015625 disk.drbd2080.out‚0( proc.memory.availableMB‚ 1309.2109375 disk.nvme0n1.in‚ 162.18359375& disk.nvme0n1.out‚23549.35791015625›=““Ç‚(:ˆ Ê—åÔÀã µ gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚131.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ô þbÈ‚(:× Ê—åÔäÒò¶ disk.drbd2265.out‚0 network.sent‚ 7.369544e+06 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ memory_percent‚92.00770854949951 network.recv‚ 1.155452e+06" disk.nvme0c0n1.in‚ 162.18359375 disk.rbd0.in‚0 disk.dm-1.out‚ 120.1015625 disk.sda2.in‚0# proc.memory.rssMB‚ 3040.47265625) proc.memory.percent‚18.55757236480713 proc.cpu.threads‚17 disk.drbd2265.in‚0# disk.dm-0.out‚23594.28759765625 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0& disk.nvme0n1.out‚23594.28759765625 disk.rbd0.out‚0$ disk./.usageGB‚274.6437530517578 disk.dm-0.in‚ 162.18359375 disk.drbd2080.in‚0 disk.sda.out‚ 120.1015625 disk.sda2.out‚0 cpu‚25.226294333597433( disk.nvme0c0n1.out‚23594.28759765625) disk./.usagePercent‚18.43157270876124 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 162.18359375) proc.memory.availableMB‚ 1309.45703125 disk.sda3.out‚ 120.1015625›óÓm2É- _wandbruntime‚3289  _runtime‚328908”ŒÉÊà _step‚266 ' lr-AdamW/pg1‚0.00018650621195779799 ' lr-AdamW/pg2‚0.00018650621195779799 trainer/global_step‚3299 % _timestamp‚1.7884313110616624e+09 _runtime‚3289.416727451lë=¾ÎËÈ ' lr-AdamW/pg1‚0.00018650621195779799 ' lr-AdamW/pg2‚0.00018650621195779799 trainer/global_step‚3299 % _timestamp‚1.7884313110616624e+09 _runtime‚3289.416727451 _step‚266Šjx²2Ì- _wandbruntime‚3289  _runtime‚3289 ÿÓ:þ'ÍøO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚444777992996197 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445260633812992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25768263 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884313110624084e+09 _runtime‚3289.417707922 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452296246519296 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70806452 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60825416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445260660204544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452296246519296 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9773184615936 trainer/global_step‚3299 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445260660204544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70806452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7035586314752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129515717 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58709259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58709259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70783347 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452293114734592 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390272329971200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437754253747200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35056590 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58709265 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7032480921600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7023713821029 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7032480921600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35056587 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129492606 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452293114734592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚563 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9773127503872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58709265 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7032480921600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7023713821029 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7032480921600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35056587 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129492606 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452293114734592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚563 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9773127503872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚444777992996197 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445260633812992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25768263 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884313110624084e+09 _runtime‚3289.417707922 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452296246519296 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70806452 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60825416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445260660204544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452296246519296 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9773184615936 trainer/global_step‚3299 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445260660204544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878‹üú@ÿ2Ï- _wandbruntime‚3290  _runtime‚3290ý}¸¹±Ð‚N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445392154292736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380611641465856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70804437 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70827568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70804437 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129530980 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58726549 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452426721136640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380611613331456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7022719145209 % _timestamp‚1.7884313119646041e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445392127901184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129554117 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437883650748416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7037698641920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60843142 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60842601 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452429852934656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7034593235456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58726543 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445392154292736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390387745125888 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7037698641920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58726549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9776103660É\,³W032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437883676176384 _runtime‚3290.320661439 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7034593235456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70827568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129530980 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25776299 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7025822614337 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452429852934656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129554117 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58726543 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9776046561280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445392127901184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35066843 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390387659892736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445392127901184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35066843 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390387659892736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452426721136640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380611613331456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7022719145209 % _timestamp‚1.7884313119646041e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445392127901184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129554117 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437883650748416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7037698641920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60843142 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60842601 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452429852934656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7034593235456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58726543 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445392154292736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390387745125888 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7037698641920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58726549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9776103660032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437883676176384 _runtime‚3290.320661439 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7034593235456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70827568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129530980 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25776299 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7025822614337 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452429852934656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129554117 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58726543 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9776046561280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 Eý'ß÷O VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458976581035520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7138111028224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61718762 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61718189 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396039070320128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458973449250816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35569240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59573465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26149522 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131414658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458976581035520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131437769 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396038985073664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71864304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7141216421376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444224103808000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71841199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386119745673728 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9919296512000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59573459 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26148952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451835338222592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59573459 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9919239399936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71864304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458973449250816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚570 _step‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35569237 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59573465 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 $ _timestamp‚1.788431358100966e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7129167067237 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111ÂóË&^àüO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 R>DezÄ—+¤'viceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458973449250816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35569240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59573465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26149522 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131414658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458976581035520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131437769 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396038985073664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71864304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7141216421376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444224103808000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71841199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386119745673728 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9919296512000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59573459 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26148952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451835338222592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59573459 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9919239399936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71864304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458973449250816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚570 _step‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35569237 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59573465 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 $ _timestamp‚1.788431358100966e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7129167067237 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458976581035520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7138111028224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61718762 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61718189 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396039070320128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552÷ˆs2á- _wandbruntime‚3337  _runtime‚3337B~ÛÃ'â‚N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451481673200633 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459110187450880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459107055652864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 trainer/global_step‚3349 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396154314116608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59590743 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451966858702336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35579490 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131453032 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131476169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386232184672256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9922157578752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451966832310784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7143328748544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59590749 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451966832310784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7131275860545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444353500809216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71885420 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451966858702336 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131453032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7128172391417 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61736116 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7140223342080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71885420 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9922214677504 % _timestamp‚1.7884313589962995e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59590749 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71862289 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7143328748544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71862289 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 214320773122®Ù 'ã‡N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131453032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7128172391417 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61736116 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7140223342080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71885420 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9922214677504 % _timestamp‚1.7884313589962995e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59590749 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71862289 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7143328748544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71862289 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451481673200633 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459110187450880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459107055652864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 trainer/global_step‚3349 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396154314116608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59590743 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451966858702336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35579490 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131453032 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131476169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386232184672256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9922157578752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451966832310784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7143328748544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59590749 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451966832310784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7131275860545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444353500809216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71885420 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451966858702336 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329kƒ#¬2ä- _wandbruntime‚3337  _runtime‚3337øý”^]å× 1loss/SRqqt1/detection_loss‚1.2145174344707477 1loss/FBt4/assignment_loss‚0.25171655533250004 /loss/FRt4/detection_loss‚3.1148959506641734 _runtime‚3337.352969575 1loss/FBt1/assignment_loss‚0.09017449533739273 .loss/FRt3/detection_loss‚2.221313125345596 .loss/FBt2/detection_loss‚2.977875118905848 _step‚273 1loss/SRqqt4/detection_loss‚2.5197827499359846 -loss/FBt3/detection_loss‚2.58501515686512 /loss/FBt4/detection_loss‚1.8825181417167187 %loss/total_loss‚47.13335999108329 /loss/FRt2/detection_loss‚2.6816483785708747 0loss/FRt1/assignment_loss‚2.1501218311822234 0loss/FRt4/assignment_loss‚4.3492043885317715 2loss/SRqqt3/assignment_loss‚1.5510171146965341 1loss/FBt2/assignment_loss‚0.23774761498299313 /loss/FRt1/detection_loss‚1.8628057849718564 trainer/global_step‚3349 2loss/SRqqt1/assignment_loss‚1.5261376309453656 2loss/SRqqt2/assignment_loss‚2.4583466136455536 1loss/FBt3/assignment_loss‚0.07425752804083459 1loss/SRqqt3/detection_loss‚1.2175416048420102 epoch‚2 /loss/FRt2/assignment_loss‚4.988018989562988 2loss/SRqqt4/assignment_loss‚1.5537188472226262 /loss/FBt1/detection_loss‚0.9787699760248264 % _timestamp‚1.7884313589980009e+09 0loss/FRt3/assignment_loss‚2.0830762788306836 0loss/SRqqt2/detection_loss‚2.563138680458069ŠÔuUæÜ 1loss/SRqqt4/detection_loss‚2.5197827499359846 -loss/FBt3/detection_loss‚2.58501515686512 /loss/FBt4/detection_loss‚1.8825181417167187 %loss/total_loss‚47.13335999108329 /loss/FRt2/detection_loss‚2.6816483785708747 0loss/FRt1/assignment_loss‚2.1501218311822234 0loss/FRt4/assignment_loss‚4.3492043885317715 2loss/SRqqt3/assignment_loss‚1.5510171146965341 1loss/FBt2/assignment_loss‚0.23774761498299313 /loss/FRt1/detection_loss‚1.8628057849718564 trainer/global_step‚3349 2loss/SRqqt1/assignment_loss‚1.5261376309453656 2loss/SRqqt2/assignment_loss‚2.4583466136455536 1loss/FBt3/assignment_loss‚0.07425752804083459 1loss/SRqqt3/detection_loss‚1.2175416048420102 epoch‚2 /loss/FRt2/assignment_loss‚4.988018989562988 2loss/SRqqt4/assignment_loss‚1.5537188472226262 /loss/FBt1/detection_loss‚0.9787699760248264 % _timestamp‚1.7884313589980009e+09 0loss/FRt3/assignment_loss‚2.0830762788306836 0loss/SRqqt2/detection_loss‚2.563138680458069 1loss/SRqqt1/detection_loss‚1.2145174344707477 1loss/FBt4/assignment_loãÁwM ss‚0.25171655533250004 /loss/FRt4/detection_loss‚3.1148959506641734 _runtime‚3337.352969575 1loss/FBt1/assignment_loss‚0.09017449533739273 .loss/FRt3/detection_loss‚2.221313125345596 .loss/FBt2/detection_loss‚2.977875118905848 _step‚273‘ùß_ßžç‚(:“ †˜åÔ øž¶ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚308.951' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.790200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–˜<>Yè‚(:Î †˜åÔÆÄ¥· disk.rbd0.in‚0 network.sent‚ 7.495572e+06 disk.dm-1.out‚122.1875 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.83984375$ memory_percent‚92.01014041900635) proc.memory.availableMB‚ 1309.05859375) proc.memory.percent‚18.55759620666504 disk.sda2.out‚0& disk.nvme0n1.out‚23765.97900390625 proc.cpu.threads‚17 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda.in‚0 disk.sda.out‚122.1875* disk./.usagePercent‚18.433860136079677( disk.nvme0c0n1.out‚23765.97900390625 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.18359375 network.recv‚ 1.173951e+06 disk.dm-1.in‚0 disk.sda3.out‚122.1875 cpu‚25.223075833991224 disk.rbd1.in‚0# disk.dm-0.out‚23765.97900390625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 3040.4765625 disk.drbd2265.in‚0$ disk./.usageGB‚274.6778373718262 disk.sda2.in‚0 disk.dm-0.in‚ 162.18359375 disk.sda3.in‚0 disk.drbd1716.out‚0¾er2é‚(:’ •˜åÔ®Õ“µ gpu.0.gpu‚68.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.548' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.50959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤”xÊMê‚(: •˜åÔ뻶 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 3040.4765625* disk./.usagePercent‚18.435245136839917 disk.dm-1.in‚0 disk.sda3.out‚122.75) proc.memory.availableMB‚ 1309.05859375 cpu‚25.222321917718613 disk.nvme0n1.in‚ 162.18359375 disk.sda2.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚122.75& disk.nvme0c0n1.out‚23829.494140625 network.recv‚ 1.181032e+06 disk.drbd2266.out‚0! disk.dm-0.out‚23829.494140625 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2265.out‚0$ disk./.usageGB‚274.6984748840332 disk.drbd2080.out‚0 disk.drbd2265.in‚0$ memory_percent‚92.01014041900635 network.sent‚ 7.604915e+06$ disk.nvme0n1.out‚23829.494140625 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.sda.out‚122.75) proc.memory.percent‚18.55759620666504 disk.rbd1.in‚0 disk.dm-0.in‚ 162.18359375 proc.cpu.threads‚17© }àë‚(:’ ¤˜åÔ¿ä²µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚258.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.77759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚%'=dì‚(:Ù ¤˜åÔ𑇷 disk.nvme0n1.in‚ 162.18359375 disk.sda2.in‚0 disk.dm-1.out‚ 123.22265625" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375# disk.dm-0.out‚23874.96923828125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚23874.96923828125$ memory_percent‚92.01016426086426 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 cpu‚25.221552637829767 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 3040.4765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 123.22265625* disk./.usagePercent‚18.435373396614573 disk.drbd2266.in‚0 disk.drbd2265.out‚0$ disk./.usageGB‚274.7003860473633 disk.drbd2266.out‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 1309.0546875& disk.nvme0n1.out‚23874.96923828125 disk.rbd1.out‚ 1.83984375 disk.sda.in‚0 disk.drbd2265.in‚0) proc.memory.percent‚18.55759620666504 network.sent‚ 7.610173e+06 network.recv‚ 1.184426e+06 disk.rbd0.in‚0 disk.sda3.out‚ 123.22265625 disk.drbd1716.out‚0x¼v2í- _wandbruntime‚3383  _runtime‚33834›ŸÉîà ' lr-AdamW/pg1‚0.00018479548540267841 ' lr-AdamW/pg2‚0.00018479548540267841 trainer/global_step‚3399 % _timestamp‚1.7884314055496163e+09 _runtime‚3383.904678346 _step‚274,ñ“rÎïÈ % _timestamp‚1.7884314055496163e+09 _runtime‚3383.904678346 _step‚274 ' lr-AdamW/pg1‚0.00018479548540267841 ' lr-AdamW/pg2‚0.00018479548540267841 trainer/global_step‚3399’‡”÷2ð- _wandbruntime‚3383  _runtime‚3383^uú0ÿ'ñùO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465653783767040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60437665 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7243741134848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚3399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60437659 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7243741134848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7246846528000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133359821 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72922156 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7246846528000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62611270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401805801302528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465656915551744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10065484288000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458410069023744 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391740317014528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36081887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26529950 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465653783767040 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7231516844525 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60437659 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391740288880128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458410069023744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7234620313445 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72899051 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457928599610213 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚450693953868800 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302Y»>7(òþO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72899051 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457928599610213 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚450693953868800 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465653783767040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60437665 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7243741134848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚3399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60437659 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7243741134848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7246846528000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133359821 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72922156 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7246846528000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62611270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401805801302528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465656915551744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10065484288000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458410069023744 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391740317014528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36081887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26529950 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465653783767040 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 401694720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7231516844525 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60437659 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391740288880128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458410069023744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7234620313445 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302“ÃÐN¥2ó- _wandbruntime‚3384  _runtime‚3384u ,xô„N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133398221 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72943272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60454949 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72920141 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚3399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133398221 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458541563111936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458541536720384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133375084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391852699744256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465790521967104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458541536720384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26536824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7233625637625 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60454943 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60454949 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62629510 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401921050401280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465787390169088 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458056976507641 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7248958855168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62628964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450823376297984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚276 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133375084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72920141 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465790521967104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60454943 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10068407755776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚3384.810725998 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7245853448704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133398221 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72943272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60454949 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72920141 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚3399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133398221 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458541563111936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 401694720 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458541536720384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133375084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391852699744256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465790521967104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458541536720384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26536824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3275873280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7233625637625 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60454943 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60454949 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62629510 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401921050401280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465787390169088 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458056976507641 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7248958855168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62628964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450823376297984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚276 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133375084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72920141 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465790521967104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60454943 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10068407755776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚3384.810725998”_Iåz2ö- _wandbruntime‚3384  _runtime‚3384Ü‹¦X÷Ò _runtime‚3384.811898481 1loss/FBt4/assignment_loss‚0.36265859184774085 0loss/FBt1/assignment_loss‚0.1957714975786051 /loss/FRt3/detection_loss‚2.0073539969325065 1loss/SRqqt1/detection_loss‚1.4099246845094127 2loss/SRqqt3/assignment_loss‚1.1754831238649786 _step‚277 1loss/FBt3/assignment_loss‚0.15556165200409436 trainer/global_step‚3399 /loss/FBt4/detection_loss‚2.2012768055383978 %loss/total_loss‚48.20572202857129 epoch‚2 0loss/FRt1/assignment_loss‚1.7147500993950027 /loss/FRt3/assignment_loss‚2.266227912902832 0loss/SRqqt2/assignment_loss‚2.19228768547376 /loss/FBt1/detection_loss‚0.8662090888107891 .loss/FRt2/detection_loss‚2.950619239073533 .loss/FRt4/detection_loss‚3.044223283727964 0loss/SRqqt2/detection_loss‚3.349792476495107 1loss/FBt2/assignment_loss‚0.22128037281423582 /loss/FRt1/detection_loss‚1.8579614352700966 /loss/FRt2/assignment_loss‚4.410607727674337 /loss/FRt4/assignment_loss‚4.811649272839229 2loss/SRqqt1/assignment_loss‚1.2113373747178868 2loss/SRqqt4/assignment_loss‚2.0349369432777165 1loss/SRqqt4/detection_loss‚2.8474882885813715 .loss/FBt2/detection_loss‚3.599132831280048 1loss/SRqqt3/detection_loss‚1.2854610262677455 .loss/FBt3/detection_loss‚2.033726617693901 % _timestamp‚1.7884314064569566e+09b¢sr]ø× 2loss/SRqqt3/assignment_loss‚1.1754831238649786 _step‚277 1loss/FBt3/assignment_loss‚0.15556165200409436 trainer/global_step‚3399 /loss/FBt4/detection_loss‚2.2012768055383978 %loss/total_loss‚48.20572202857129 epoch‚2 0loss/FRt1/assignment_loss‚1.7147500993950027 /loss/FRt3/assignment_loss‚2.266227912902832 0loss/SRqqt2/assignment_loss‚2.19228768547376 /loss/FBt1/detection_loss‚0.8662090888107891 .loss/FRt2/detection_loss‚2.950619239073533 .loss/FRt4/detection_loss‚3.044223283727964 0loss/SRqqt2/detection_loss‚3.349792476495107 1loss/FBt2/assignment_loss‚0.22128037281423582 /loss/FRt1/detection_loss‚1.8579614352700966 /loss/FRt2/assignment_loss‚4.410607727674337 /loss/FRt4/assignment_loss‚4.811649272839229 2loss/SRqqt1/assignment_loss‚1.2113373747178868 2loss/SRqqt4/assignment_loss‚2.0349369432777165 1loss/SRqqt4/detection_loss‚2.8474882885813715 .loss/FBt2/detection_loss‚3.599132831280048 1loss/SRqqt3/detection_loss‚1.2854610262677455 .loss/FBt3/detection_loss‚2.033726617693901 % _timestamp‚1.7884314064569566e+09 _runtime‚3384.811898481 1loss/FBt4/assignment_loss‚0.36265859184774085 0loss/FBt1/assignment_loss‚0.1957714975786051 /loss/FRt3/detection_loss‚2.0073539969325065 1loss/SRqqt1/detection_loss‚1.4099246845094127•Њ”Ø“ù‚(:ˆ ³˜åÔ°Ÿ­µ gpu.0.gpu‚63.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.811' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7622 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CYOªcú‚(:Ø ³˜åÔÛÆø¶ disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 162.18359375 disk.dm-0.in‚ 162.18359375 disk.sda3.out‚ 123.76953125 disk.sda2.out‚0 disk.rbd1.out‚ 1.83984375) proc.memory.availableMB‚ 1309.01953125 disk.rbd0.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23911.16552734375# disk.dm-0.out‚23911.16552734375 disk.drbd1716.in‚0 disk.sda.out‚ 123.76953125 disk.dm-1.out‚ 123.76953125 disk.drbd2080.in‚0" proc.memory.rssMB‚ 3040.4765625 network.recv‚ 1.189254e+06* disk./.usagePercent‚18.437229963252882 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0) proc.memory.percent‚18.55759620666504 cpu‚25.2208011023118$ disk./.usageGB‚274.7280502319336 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 7.616163e+06 disk.nvme0n1.in‚ 162.18359375& disk.nvme0n1.out‚23911.16552734375 disk.sda1.in‚0 disk.drbd2080.out‚0$ memory_percent‚92.01037883758545 proc.cpu.threads‚17Á̽Ҟû‚(:“ ˜åÔ؆µ gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚278.103' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.620599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rý¢`ü‚(:Õ ˜åÔ¨‘Ô¶& disk.nvme0n1.out‚23936.77880859375 disk.sda.out‚ 124.109375" proc.memory.rssMB‚ 3040.4765625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 network.sent‚ 7.724772e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.83984375 disk.dm-0.in‚ 162.18359375 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 162.18359375( disk.nvme0c0n1.out‚23936.77880859375# disk.dm-0.out‚23936.77880859375 disk.sda3.out‚ 124.109375 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.55759620666504 disk.dm-1.out‚ 124.109375 disk.rbd1.in‚0$ memory_percent‚92.01037883758545 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 cpu‚25.220058659436614 disk.drbd2080.in‚0% disk./.usageGB‚272.50515365600586 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1309.01953125 network.recv‚ 1.195437e+06* disk./.usagePercent‚18.288049508907918" disk.nvme0c0n1.in‚ 162.18359375 û¡Vý‚(:’ јåÔùŸŽµ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚258.599' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.71979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€îØ¡Sþ‚(:È јåÔ›×”¸ disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 198.09765625 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.83984375" disk.nvme0c0n1.out‚ 24044.78125 disk.sda1.out‚0 disk.drbd1716.out‚0$ memory_percent‚92.01047420501709) proc.memory.percent‚18.55759620666504" disk.nvme0c0n1.in‚ 198.09765625" proc.memory.rssMB‚ 3040.4765625 network.recv‚ 1.200365e+06 disk.sda.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1309.00390625 cpu‚25.219295496441184 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.304678734302026 disk.drbd2265.out‚0 disk.dm-0.out‚ 24044.78125 disk.sda3.out‚ 124.86328125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 198.09765625 disk.sda2.out‚0 disk.drbd2266.out‚0$ disk./.usageGB‚272.7529411315918 disk.sda.out‚ 124.86328125 disk.dm-1.out‚ 124.86328125 disk.drbd1716.in‚0 network.sent‚ 7.731157e+06 disk.nvme0n1.out‚ 24044.78125‰^TÀ2ÿ- _wandbruntime‚3430  _runtime‚3430u!›ÊÇ€Á & lr-AdamW/pg1‚0.0001839142395661159 & lr-AdamW/pg2‚0.0001839142395661159 trainer/global_step‚3449 % _timestamp‚1.7884314523215182e+09 _runtime‚3430.676578654 _step‚278œ¦”ÌÆ trainer/global_step‚3449 % _timestamp‚1.7884314523215182e+09 _runtime‚3430.676578654 _step‚278 & lr-AdamW/pg1‚0.0001839142395661159 & lr-AdamW/pg2‚0.0001839142395661159–î5Ðs2‚- _wandbruntime‚3430  _runtime‚3430’Ø þ'ƒøO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63503678 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7352476634624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73956903 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10211286655488 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7340073559653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73980008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61301865 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61301865 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚472334118283264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397360832086528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464984747041792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36594540 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407572206085632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464500774020333 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36594537 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457163829357568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 # _timestamp‚1.78843145232228e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚472337250067968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61301859 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73980008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7349371241472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135281873 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚3430.677587939 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61301859 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407572118742016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚279 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26909141 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457163803929600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7336970090733 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚3449ŸzL‰(„ýO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407572118742016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479245 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚279 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26909141 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457163803929600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7336970090733 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚3449 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63503678 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ^LDeviceSta>ì±Û"tsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7352476634624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73956903 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10211286655488 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7340073559653 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73980008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61301865 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61301865 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚472334118283264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397360832086528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464984747041792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36594540 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407572206085632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464500774020333 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36594537 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457163829357568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 # _timestamp‚1.78843145232228e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚472337250067968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61301859 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73980008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7349371241472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135281873 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚3430.677587939 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61301859—ÅÿÚ{2…- _wandbruntime‚3431  _runtime‚3431¸Zà› '†„N _runtime‚3431.564645267 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7351483555328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61319143 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407687460173824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10214276419072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚472470856483328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464632279814649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73977993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10214217223168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61319143 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚465116241129984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26917504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74001124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63522297 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚465116267521536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397473271085056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7342182352961 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7354588961792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397473242950656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464635408711745 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚457293200930816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135297136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚457293226358784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135320273 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚3449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚472467724685312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61319149 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87330304 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74001124 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7354588961792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73977993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63521731 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚465116241129984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚563 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407687547504128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135297136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135320273 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61319149 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚472470856483328 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7354588961792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73977993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63521731 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚465116241129984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚563 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407687547504128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135297136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135320273 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61319149 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚472470856483328 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464632279814649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73977993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10214217223168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61319143 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚465116241129984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26917504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74001124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63522297 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚465116267521536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397473271085056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7342182352961 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7354588961792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397473242950656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464635408711745 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚457293200930816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135297136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚457293226358784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135320273 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚3449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚472467724685312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61319149 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87330304˜G –2ˆ- _wandbruntime‚3431  _runtime‚3431V¬˜ÊZ‰Ô 1loss/FBt4/assignment_loss‚0.16645289581677594 /loss/FBt2/detection_loss‚2.4977196902036667 _runtime‚3431.566037154 _step‚281 /loss/FRt1/assignment_loss‚2.386052973453815 2loss/SRqqt1/assignment_loss‚1.3874404751404654 /loss/FRt4/detection_loss‚3.0059045976208103 /loss/FBt3/detection_loss‚2.8512442857027054 /loss/FRt2/assignment_loss‚5.002556285858154 /loss/FRt3/detection_loss‚2.1662436098587223 -loss/FBt1/detection_loss‚1.14266646951437 /loss/FRt4/assignment_loss‚4.678288305959394 1loss/FBt1/assignment_loss‚0.06922556481062074 /loss/FBt4/detection_loss‚1.4453846919200113 2loss/SRqqt2/assignment_loss‚2.0708373992339424 2loss/SRqqt3/assignment_loss‚1.3365587220666395 1loss/SRqqt3/detection_loss‚1.2258498826246151 0loss/FRt3/assignment_loss‚1.8317236347896297 1loss/FBt3/assignment_loss‚0.10770800802856684 .loss/FRt2/detection_loss‚2.516745300292969 1loss/SRqqt1/detection_loss‚1.1016025508288294 % _timestamp‚1.7884314532107959e+09 2loss/SRqqt4/assignment_loss‚2.0402425026351754 %loss/total_loss‚47.16719256357762 epoch‚2 trainer/global_step‚3449 0loss/FBt2/assignment_loss‚0.4232908329304319 /loss/FRt1/detection_loss‚2.0075751930475234 /loss/SRqqt2/detection_loss‚3.18108731508255 1loss/SRqqt4/detection_loss‚2.5247913761572405£ _ŠÙ 1loss/FBt4/assignment_loss‚0.16645289581677594 /loss/FBt2/detection_loss‚2.4977196902036667 _runtime‚3431.566037154 _step‚281 /loss/FRt1/assignment_loss‚2.386052973453815 2loss/SRqqt1/assignment_loss‚1.3874404751404654 /loss/FRt4/detection_loss‚3.0059045976208103 /loss/FBt3/detection_loss‚2.8512442857027054 /loss/FRt2/assignment_loss‚5.002556285858154 /loss/FRt3/detection_loss‚2.1662436098587223 -loss/FBt1/detection_loss‚1.14266646951437 /loss/FRt4/assignment_loss‚4.678288305959394 1loss/FBt1/assignment_loss‚0.06922556481062074 /loss/FBt4/detection_loss‚1.4453846919200113 2loss/SRqqt2/assignment_loss‚2.0708373992339424 2loss/SRqqt3/assignment_loss‚1.3365587220666395 1loss/SRqqt3/detection_loss‚1.2258498826246151 0loss/FRt3/assignment_loss‚1.8317236347896297 1loss/FBt3/assignment_loss‚0.10770800802856684 .loss/FRt2/detection_loss‚2.516745300292969 1loss/SRqqt1/detection_loss‚1.1016025508288294 % _timestamp‚1.7884314532107959e+09 2loss/SRqqt4/assignment_loss‚2.0402425026351754 %loss/total_loss‚47.16719256357762 epoch‚2 trainer/global_step‚3449 0loss/FBt2/assignment_loss‚0.4232908329304319 /loss/FRt1/detection_loss‚2.0075751930475234 /loss/SRqqt2/detection_loss‚3.18108731508255 1loss/SRqqt4/detection_loss‚2.5247913761572405™8;%fž‹‚(:“ à˜åÔ¡å²µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.328' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.465599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@<ÚÈŒ‚(:Ê à˜åԢ㾷) proc.memory.percent‚18.55759620666504% disk./.usageGB‚273.00242614746094 disk.nvme0n1.in‚ 204.2578125 disk.sda.out‚ 125.51171875 proc.cpu.threads‚17" disk.nvme0n1.out‚ 24209.2890625 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 204.2578125 disk.dm-1.in‚0 disk.dm-0.out‚ 24209.2890625 disk.drbd2080.out‚0 cpu‚25.21849949028782 network.sent‚ 7.736418eVÀŒªC+06 network.recv‚ 1.203507e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1308.7265625 disk.drbd2266.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 3040.4765625 disk.sda3.in‚0 disk.sda3.out‚ 125.51171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 125.51171875 disk.rbd1.out‚ 1.83984375$ disk.nvme0c0n1.out‚ 24209.2890625* disk./.usagePercent‚18.321421883048878 disk.rbd0.out‚0 disk.dm-0.in‚ 204.2578125$ memory_percent‚92.01216697692871*Κ“‚(:ˆ ï˜åÔÐáí´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚255.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„n![Ž‚(:Ð ï˜åÔò†Å¸ disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 126.21875 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 210.29296875 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 1308.6953125 proc.cpu.threads‚17 network.sent‚ 7.844994e+06 disk.drbd1716.out‚0 disk.drbd2265.in‚0) disk./.usagePercent‚18.35497730257682 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚24516.95654296875 disk.drbd2080.in‚0) proc.memory.percent‚18.55759620666504& disk.nvme0n1.out‚24516.95654296875 disk.rbd1.out‚ 1.83984375# disk.dm-0.out‚24516.95654296875 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 126.21875 network.recv‚ 1.209942e+06 disk.nvme0n1.in‚ 210.29296875 disk.drbd1716.in‚0" proc.memory.rssMB‚ 3040.4765625 cpu‚25.217756598828867 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0$ memory_percent‚92.01235771179199% disk./.usageGB‚273.50242614746094 disk.dm-1.out‚ 126.21875" disk.nvme0c0n1.in‚ 210.29296875>>ÚBž‚(:“ þ˜åÔϱœµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚162.921' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.584199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ц·Ìb‚(:× þ˜åÔ™£· disk.rbd0.out‚0( disk.nvme0c0n1.out‚24764.15576171875 disk.dm-0.in‚ 215.78515625 disk.sda3.in‚0" proc.memory.rssMB‚ 3040.4765625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda.out‚ 126.54296875 disk.sda2.out‚0* disk./.usagePercent‚18.354951701823396 disk.drbd2080.in‚0$ disk./.usageGB‚273.5020446777344 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 cpu‚25.2170546685073 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd2265.in‚0& disk.nvme0n1.out‚24764.15576171875 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 215.78515625 disk.sda3.out‚ 126.54296875 disk.drbd1716.in‚0( proc.memory.availableMB‚ 1308.9453125) proc.memory.percent‚18.55759620666504 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 215.78515625 disk.dm-1.out‚ 126.54296875# disk.dm-0.out‚24764.15576171875 disk.drbd1716.out‚0 disk.drbd2080.out‚0$ memory_percent‚92.01083183288574 network.sent‚ 7.850903e+06 network.recv‚ 1.213779e+06hlC¨2‘- _wandbruntime‚3477  _runtime‚3477òº+É’Ã ' lr-AdamW/pg1‚0.00018301598236282829 ' lr-AdamW/pg2‚0.00018301598236282829 trainer/global_step‚3499 % _timestamp‚1.7884314988918695e+09 _runtime‚3477.246892128 _step‚282U Î“È ' lr-AdamW/pg1‚0.00018301598236282829 ' lr-AdamW/pg2‚0.00018301598236282829 trainer/global_step‚3499 % _timestamp‚1.7884314988918695e+09 _runtime‚3477.246892128 _step‚282šôõèw2”- _wandbruntime‚3477  _runtime‚3477S…nØ(•úO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚479017584584192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27291262 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7458106741248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3477.248058439 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7445526805861 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚471079206224229 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75014755 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7455001348096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479014452799488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10357878835200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 trainer/global_step‚3499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62166059 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463633679418368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463633653990400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64397890 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7458106741248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27290703 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚402981403427328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚471559477842944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚471559477842944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚479017584584192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27291262 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7458106741248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3477.248058439 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7445526805861 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚471079206224229 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75014755 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7455001348096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479014452799488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10357878835200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83149312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 trainer/global_step‚3499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62166059 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463633679418368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463633653990400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64397890 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7458106741248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27290703 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚402981403427328 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64415692 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚479151190999552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚471690971931136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403093786157056 _runtime‚3478.133248709 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚471690971931136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62183349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75035845 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137242325 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471210712018753 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62183349 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10360801894400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37117440 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62183343 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7460219068416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7457113661952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚3499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137219188 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479151190999552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413454671187456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75058976 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10360856896000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64416238 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 % _timestamp‚1.7884314997775342e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479148059201536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚471690945539584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚284 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37117443 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413454588051456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7444532130041 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚471690945539584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on äþÈ_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463763076419584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092•*÷m'™‰N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62183349 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10360801894400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37117440 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62183343 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7460219068416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7457113661952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚3499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137219188 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479151190999552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413454671187456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75058976 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10360856896000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64416238 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 % _timestamp‚1.7884314997775342e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479148059201536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚471690945539584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚284 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37117443 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413454588051456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7444532130041 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚471690945539584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463763076419584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693092 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137219188 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27298252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64415692 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚479151190999552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚471690971931136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403093786157056 _runtime‚3478.133248709 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚471690971931136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62183349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75035845 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137242325 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471210712018753 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0œ²¯2š- _wandbruntime‚3478  _runtime‚3478Ãç±X›Ò /loss/FRt4/assignment_loss‚4.554884866873423 1loss/FBt1/assignment_loss‚0.07504138670184952 _runtime‚3478.134546819 1loss/SRqqt4/assignment_loss‚2.001015661393895 0loss/FBt4/assignment_loss‚0.3158343334287076 1loss/SRqqt1/detection_loss‚1.5275522325251063 /loss/FBt2/detection_loss‚2.8086406062631046 0loss/FBt2/assignment_loss‚0.2730506585645166 0loss/SRqqt4/detection_loss‚2.424708953675102 %loss/total_loss‚46.17582591032953 2loss/SRqqt2/assignment_loss‚1.9640929418946467 2loss/SRqqt3/assignment_loss‚1.2666541998429846 0loss/SRqqt2/detection_loss‚2.581003196777836 epoch‚2 % _timestamp‚1.7884314997795737e+09 2loss/SRqqt1/assignment_loss‚1.1369665625694572 /loss/FRt1/detection_loss‚1.7166222008211272 /loss/FBt3/detection_loss‚2.6975430931363786 /loss/FRt1/assignment_loss‚2.727012515919549 0loss/FBt3/assignment_loss‚0.2291671844260301 .loss/FRt2/detection_loss‚2.507283680366747 .loss/FRt4/detection_loss‚3.100035238265991 /loss/FBt1/detection_loss‚0.8420317753246336 /loss/FBt4/detection_loss‚1.4940876531781573 0loss/FRt3/assignment_loss‚1.9809466832213931 trainer/global_step‚3499 /loss/FRt2/assignment_loss‚4.572605306451971 /loss/FRt3/detection_loss‚2.0379959590457104 1loss/SRqqt3/detection_loss‚1.3410490196612146 _step‚285Ö¦Ž]œ× 1loss/SRqqt4/assignment_loss‚2.001015661393895 0loss/FBt4/assignment_loss‚0.3158343334287076 1loss/SRqqt1/detection_loss‚1.5275522325251063 /loss/FBt2/detection_loss‚2.8086406062631046 0loss/FBt2/assignment_loss‚0.2730506585645166 0loss/SRqqt4/detection_loss‚2.424708953675102 %loss/total_loss‚46.17582591032953 2loss/SRqqt2/assignment_loss‚1.9640929418946467 2loss/SRqqt3/assignment_loss‚1.2666541998429846 0loss/SRqqt2/detection_loss‚2.581003196777836 epoch‚2 % _timestamp‚1.7884314997795737e+09 2loss/SRqqt1/assignment_loss‚1.1369665625694572 /loss/FRt1/detection_loss‚1.7166222008211272 /loss/FBt3/detection_loss‚2.6975430931363786 /loss/FRt1/assignment_loss‚2.727012515919549 0loss/FBt3/assignment_loss‚0.2291671844260301 .loss/FRt2/detection_loss‚2.507283680366747 .loss/FRt4/detection_loss‚3.100035238265991 /loss/FBt1/detection_loss‚0.8420317753246336 /loss/FBt4/detection_loss‚1.4940876531781573 0loss/FRt3/assignment_loss‚1.9809466832213931 trainer/global_step‚3499 /loss/FRt2/assignment_loss‚4.572605306451971 /loss/FRt3/detection_loss‚2.0379959590457104 1loss/SRqqt3/detection_loss‚1.3410490196612146 _step‚285 /loss/FRt4/assignment_loss‚4.554884866873423 1loss/FBt1/assignment_loss‚0.07504138670184952 _runtime‚3478.134546819 Õ¨sž‚(:“ ™åÔÖ±µ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚195.164' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.032799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø%ëvPž‚(:Å ™åÔ‰´›¹ disk.nvme0n1.out‚ 25042.46875 disk.rbd1.in‚0 network.sent‚ 7.857173e+06 disk.sda.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 127.13671875 cpu‚25.216310910891096$ disk./.usageGB‚274.5023078918457" disk.nvme0c0n1.out‚ 25042.46875 disk.dm-0.out‚ 25042.46875 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375 network.recv‚ 1.218792e+06 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.422080205399148! disk.nvme0c0n1.in‚ 221.4921875 disk.dm-0.in‚ 221.4921875 disk.sda3.out‚ 127.13671875 disk.sda.out‚ 127.13671875 disk.drbd2265.in‚0 disk.dm-1.in‚0$ memory_percent‚92.01376438140869 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 221.4921875) proc.memory.availableMB‚ 1308.46484375 disk.sda2.out‚0" proc.memory.rssMB‚ 3040.4765625) proc.memory.percent‚18.55759620666504 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0Iõ#“Ÿ‚(:ˆ œ™åÔꎵµ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.319' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.6638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€:œ!U ‚(:Ê œ™åÔÅÖþ¶ network.sent‚ 7.965153e+06 disk.sda.out‚ 127.5390625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 227.78125 disk.sda2.in‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 3040.4765625 network.recv‚ 1.224676e+06 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚25295.44091796875 disk.drbd2080.out‚0 disk.sda.in‚0 disk.sda3.in‚0) proc.memory.percent‚18.55759620666504$ disk./.usageGB‚274.5022888183594 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 227.78125# disk.dm-0.out‚25295.44091796875 disk.sda3.out‚ 127.5390625& proc.memory.availableMB‚ 1309.15625* disk./.usagePercent‚18.422078925361475 disk.drbd2265.in‚0 disk.dm-1.out‚ 127.5390625 disk.drbd1716.in‚0# memory_percent‚92.0095443725586 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚25295.44091796875 disk.sda1.in‚0 disk.rbd1.out‚ 1.83984375 disk.dm-0.in‚ 227.78125 cpu‚25.215564582756702 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17­CP@ž¡‚(:“ «™åÔ½²ôµ gpu.0.gpu‚65.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.480399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÝSn]¢‚(:Ò «™åÔýžÍ¸ disk.dm-1.in‚0& proc.memory.availableMB‚ 1308.65625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda3.out‚ 128.30078125 disk.sda.out‚ 128.30078125) proc.memory.percent‚18.55759620666504 network.sent‚ 7.971123e+06" disk.nvme0c0n1.in‚ 233.73046875 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 233.73046875 disk.sda2.in‚0 cpu‚25.21483820517071" disk.dm-0.out‚25560.8935546875 disk.drbd2080.out‚0# memory_percent‚92.0125961303711 proc.cpu.threads‚17 disk.drbd1716.out‚0% disk.nvme0n1.out‚25560.8935546875 disk.dm-0.in‚ 233.73046875 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.recv‚ 1.229137e+06$ disk./.usageGB‚274.5022888183594' disk.nvme0c0n1.out‚25560.8935546875 disk.sda2.out‚0 disk.dm-1.out‚ 128.30078125 disk.rbd1.in‚0" proc.memory.rssMB‚ 3040.4765625* disk./.usagePercent‚18.422078925361475 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd2265.out‚0 °2£- _wandbruntime‚3523  _runtime‚3523]†É¤Ã _runtime‚3523.444145875 _step‚286 ' lr-AdamW/pg1‚0.00018210090098827044 ' lr-AdamW/pg2‚0.00018210090098827044 trainer/global_step‚3549 % _timestamp‚1.7884315450891197e+09žž­Î¥È ' lr-AdamW/pg1‚0.00018210090098827044 ' lr-AdamW/pg2‚0.00018210090098827044 trainer/global_step‚3549 % _timestamp‚1.7884315450891197e+09 _runtime‚3523.444145875 _step‚286ž‚¦o2¦- _wandbruntime‚3523  _runtime‚3523Öx$ôÿ'§ùO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139102866 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485697919100416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65291895 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76072607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65291325 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚470103529479168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485697919100416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485694787315712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478134182252544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10503845245440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7550980052069 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485694787315712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7563736847872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37619837 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408601918499328 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63030259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚3523.469303569 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10503788133376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚477654509531237 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63030265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63030259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 $ _timestamp‚1.788431545113995e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 trainer/global_step‚3549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27672055 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚470103504051200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63030265 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478134182252544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7560631454720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚477651380634349 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419105706632704 _step‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408601946633728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092Sà§ð ¨þO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131784704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63030259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57112064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 $ _timestamp‚1.788431545113995e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 trainer/global_step‚3549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27672055 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚470103504051200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63030265 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478134182252544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7560631454720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚477651380634349 ZFDeviceStatsMonitÉ2Õor.on_train_batch_start/inactive_split_bytes.all.freed‚419105706632704 _step‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408601946633728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85246464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693092 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139102866 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485697919100416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65291895 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76072607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65291325 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚470103529479168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485697919100416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485694787315712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478134182252544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10503845245440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7550980052069 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485694787315712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7563736847872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37619837 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408601918499328 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63030259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚3523.469303569 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23105 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10503788133376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚477654509531237 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63030265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ŸW «2©- _wandbruntime‚3524  _runtime‚3524aH?› 'ª„N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76093697 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7549985376249 _runtime‚3524.355540531 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚477786015325761 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37630090 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139141240 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7553088845377 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7565849175040 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76116828 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7562743768576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139164377 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63047549 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27679414 trainer/global_step‚3549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485828393717760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478265676340736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65308976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478265676340736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10506763179008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚470232901052416 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478265649949184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63047543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419221120676864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76116828 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚470232926480384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7565849175040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485831525515776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76093697"z'«‰N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485831525515776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76093697 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76093697 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7549985376249 _runtime‚3524.355540531 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85233152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚477786015325761 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37630090 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139141240 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7553088845377 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7565849175040 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76116828 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7562743768576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139164377 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63047549 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27679414 trainer/global_step‚3549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485828393717760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478265676340736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65308976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798016 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478265676340736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10506763179008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚470232901052416 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478265649949184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63047543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419221120676864 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76116828 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚470232926480384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7565849175040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 Ë› t2¬- _wandbruntime‚3524  _runtime‚3524'»X™W­Ñ 2loss/SRqqt3/assignment_loss‚1.2433018331743173 1loss/SRqqt4/assignment_loss‚1.836744592658111 1loss/SRqqt3/detection_loss‚1.3407604391309056 /loss/FBt3/detection_loss‚2.7532485886053606 trainer/global_step‚3549 _step‚289 1loss/FBt1/assignment_loss‚0.22016296797374652 % _timestamp‚1.7884315460016568e+09 0loss/FRt1/assignment_loss‚2.5447377439202934 0loss/SRqqt2/detection_loss‚3.123135362352644 .loss/FBt1/detection_loss‚0.692364951968193 .loss/FBt2/detection_loss‚2.688989613737379 _runtime‚3524.356906048 &loss/total_loss‚47.491386137643616 /loss/FRt4/assignment_loss‚4.157982433995893 2loss/SRqqt2/assignment_loss‚2.5773906395548867 /loss/FRt4/detection_loss‚2.9704413183273806 1loss/SRqqt1/detection_loss‚1.3041330473023955 2loss/SRqqt1/assignment_loss‚1.3570614880171872 1loss/SRqqt4/detection_loss‚2.4379049305404936 0loss/FBt2/assignment_loss‚0.6013781699751105 0loss/FBt3/assignment_loss‚0.2961241939548962 1loss/FBt4/assignment_loss‚0.08336784464454743 .loss/FRt1/detection_loss‚2.070964222452764 /loss/FRt2/assignment_loss‚4.599008597433567 epoch‚2 .loss/FRt2/detection_loss‚2.540463659912348 .loss/FRt3/assignment_loss‚2.40926354765892 /loss/FRt3/detection_loss‚1.8897509452700616 .loss/FBt4/detection_loss‚1.752705005082217… ?"\®Ö 2loss/SRqqt1/assignment_loss‚1.3570614880171872 1loss/SRqqt4/detection_loss‚2.4379049305404936 0loss/FBt2/assignment_loss‚0.6013781699751105 0loss/FBt3/assignment_loss‚0.2961241939548962 1loss/FBt4/assignment_loss‚0.08336784464454743 .loss/FRt1/detection_loss‚2.070964222452764 /loss/FRt2/assignment_loss‚4.599008597433567 epoch‚2 .loss/FRt2/detection_loss‚2.540463659912348 .loss/FRt3/assignment_loss‚2.40926354765892 /loss/FRt3/detection_loss‚1.8897509452700616 .loss/FBt4/detection_loss‚1.752705005082217 2loss/SRqqt3/assignment_loss‚1.2433018331743173 1loss/SRqqt4/assignment_loss‚1.836744592658111 1loss/SRqqt3/detection_loss‚1.3407604391309056 /loss/FBt3/detection_loss‚2.7532485886053606 trainer/global_step‚3549 _step‚289 1loss/FBt1/assignment_loss‚0.22016296797374652 % _timestamp‚1.7884315460016568e+09 0loss/FRt1/assignment_loss‚2.5447377439202934 0loss/SRqqt2/detection_loss‚3.123135362352644 .loss/FBt1/detection_loss‚0.692364951968193 .loss/FBt2/detection_loss‚2.688989613737379 _runtime‚3524.356906048 &loss/total_loss‚47.491386137643616 /loss/FRt4/assignment_loss‚4.157982433995893 2loss/SRqqt2/assignment_loss‚2.5773906395548867 /loss/FRt4/detection_loss‚2.9704413183273806 1loss/SRqqt1/detection_loss‚1.3041330473023955¡ž9cµ¯‚(:’ º™åԧ蠵 gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚260.545' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.10900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!NœV°‚(:Ë º™åÔп˜¸ network.sent‚ 7.977082e+06 network.recv‚ 1.233376e+06 disk.sda3.out‚ 128.66796875( proc.memory.availableMB‚ 1578.9921875 cpu‚25.214077701472473 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 128.66796875 disk.sda1.out‚0 disk.drbd1716.in‚0* disk./.usagePercent‚18.422078925361475 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0$ memory_percent‚90.36259651184082) proc.memory.percent‚16.84439182281494 disk.rbd0.in‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚25727.033203125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 239.65625 disk.dm-0.in‚ 239.65625! disk.dm-0.out‚25727.033203125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 128.66796875# proc.memory.rssMB‚ 2759.78515625 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda3.in‚0 disk.sda.in‚0$ disk./.usageGB‚274.5022888183594 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 239.65625& disk.nvme0c0n1.out‚25727.033203125 disk.drbd2080.in‚0ziÝž±‚(:“ É™åÔâõ©µ gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚287.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.510600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú29(:Ø É™åÔšµ‰· disk.dm-1.out‚ 129.10546875 disk.drbd1716.out‚0 disk.rbd0.out‚0 network.recv‚ 1.240123e+06 disk.nvme0n1.in‚ 245.56640625& disk.nvme0n1.out‚25924.37841796875 disk.drbd2266.in‚0" di_¿‰ñusk.nvme0c0n1.in‚ 245.56640625 disk.sda3.out‚ 129.10546875 disk.sda1.out‚0 disk.sda.out‚ 129.10546875 network.sent‚ 8.086055e+06# disk.dm-0.out‚25924.37841796875 disk.sda1.in‚0 disk.drbd2080.out‚0$ memory_percent‚90.42682647705078& proc.memory.availableMB‚ 1568.46875) proc.memory.percent‚16.92984104156494 cpu‚25.21333354962378 disk.sda2.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚25924.37841796875 disk.sda.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2773.78515625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0% disk./.usageGB‚276.51030349731445 disk.dm-0.in‚ 245.56640625 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd1716.in‚0* disk./.usagePercent‚18.556838475302687ÛKñ“³‚(:ˆ Ø™åÔì¿Æµ gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚263.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þűAd´‚(:Ù Ø™åÔí©· disk.rbd0.out‚0 disk.dm-1.in‚0* proc.memory.percent‚17.020153999328613 network.recv‚ 1.243541e+06" disk.nvme0c0n1.in‚ 251.90234375# disk.dm-0.out‚26267.90966796875 disk.sda1.in‚0) proc.memory.availableMB‚ 1558.88671875& disk.nvme0n1.out‚26267.90966796875( disk.nvme0c0n1.out‚26267.90966796875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 cpu‚25.212645638180295 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 network.sent‚ 8.091352e+06 disk.nvme0n1.in‚ 251.90234375 disk.sda2.in‚0 disk.dm-0.in‚ 251.90234375 disk.drbd1716.in‚0% disk./.usageGB‚276.50229263305664 disk.rbd1.out‚ 1.83984375 disk.sda3.out‚ 129.5859375 disk.sda.in‚0* disk./.usagePercent‚18.556300859480782 disk.dm-1.out‚ 129.5859375 disk.sda3.in‚0 disk.sda.out‚ 129.5859375 disk.drbd2265.out‚0# memory_percent‚90.4853105545044# proc.memory.rssMB‚ 2788.58203125 proc.cpu.threads‚17 disk.drbd2266.in‚0L% À“µ‚(:ˆ ç™åÔ÷¬˜µ gpu.0.gpu‚89.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚302.927' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.5854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)¹ìd¶‚(:Ù ç™åÔå¶& disk.nvme0n1.out‚26570.09228515625 network.recv‚ 1.248135e+06 disk.drbd1716.in‚0 disk.sda.out‚ 130.4140625 disk.rbd0.in‚0* proc.memory.percent‚17.084932327270508* disk./.usagePercent‚18.556310331759548 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚26570.09228515625# disk.dm-0.out‚26564.21728515625 disk.dm-1.out‚ 130.4140625 cpu‚25.211979047424705 disk.rbd1.out‚ 1.83984375 disk.rbd0.out‚0) proc.memory.availableMB‚ 1549.70703125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd2265.in‚0% disk./.usageGB‚276.50243377685547" disk.nvme0c0n1.in‚ 257.90234375 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 257.90234375" proc.memory.rssMB‚ 2799.1953125 network.sent‚ 8.097471e+06 disk.sda3.out‚ 130.4140625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.54133892059326 disk.dm-0.in‚ 257.90234375 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0üµ¾2·- _wandbruntime‚3570  _runtime‚3570Òxɸà _runtime‚3570.025116263 _step‚290 ' lr-AdamW/pg1‚0.00018116918614402924 ' lr-AdamW/pg2‚0.00018116918614402924 trainer/global_step‚3599 % _timestamp‚1.7884315916700883e+09F3?Î¹È ' lr-AdamW/pg1‚0.00018116918614402924 ' lr-AdamW/pg2‚0.00018116918614402924 trainer/global_step‚3599 % _timestamp‚1.7884315916700883e+09 _runtime‚3570.025116263 _step‚290¢~õùÀ2º- _wandbruntime‚3570  _runtime‚3570Š·ºë(»úO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484708860270592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28053419 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484229812838245 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚476573379539968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7666261561344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77130459 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7669366954496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77130459 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424873040478208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77153564 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚3599 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7666261561344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66185311 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 _step‚291 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141024918 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484226683941357 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66185909 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚476573354112000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63894465 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131784704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77153564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10650491428864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7653329829357 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7669366954496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚598 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28052824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59209216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27963 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484708860270592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚492375121831936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484708886662144 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27963 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484708860270592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚492375121831936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484708886662144 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77130459 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7669366954496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77130459 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424873040478208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77153564 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚3599 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7666261561344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66185311 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 _step‚291 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23105 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141024918 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484226683941357 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66185909 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚476573354112000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878631936£…4“Ú2½- _wandbruntime‚3570  _runtime‚3570é¤}¾„N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28060285 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚492511860032000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141086429 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10653475217408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚492511860032000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7668373875200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141063292 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77151549 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7655438622457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7668373875200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484840380750336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38142743 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66203575 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484840380750336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424988375921664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚414334900704256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484361318632769 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671479281664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚3599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484840354358784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10653416021504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚476702776541184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671479281664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484840354358784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66203025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424988288591360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚476702751113216 _step‚292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141086429 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63911749 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚492508728233984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7658542091585 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3570.912931699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_b c šúytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141063292 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484840380750336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424988375921664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚414334900704256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484361318632769 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671479281664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878631936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚3599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484840354358784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10653416021504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479245 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚476702776541184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671479281664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484840354358784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66203025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424988288591360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚476702751113216 _step‚292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141086429 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63911749 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚492508728233984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7658542091585 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3570.912931699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141063292 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798016 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141086429 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10653475217408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚492511860032000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7668373875200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141063292 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77151549 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7655438622457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7668373875200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484840380750336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38142743¤öÙÙ2À- _wandbruntime‚3570  _runtime‚3570’"œ™]Á× _step‚293 0loss/FRt1/assignment_loss‚2.2612515276128597 2loss/SRqqt3/assignment_loss‚1.2654156561124406 .loss/FRt2/detection_loss‚2.622932129436069 1loss/SRqqt1/detection_loss‚1.5277014233849264 /loss/FBt4/detection_loss‚1.7803778395704601 2loss/SRqqt2/assignment_loss‚2.5502874038436194 0loss/FBt1/assignment_loss‚0.2186932456278301 0loss/SRqqt4/detection_loss‚2.830264567087094 2loss/SRqqt1/assignment_loss‚1.0813680938699028 /loss/FRt3/detection_loss‚2.7635677742106575 epoch‚2 % _timestamp‚1.7884315925590339e+09 _runtime‚3570.913984115 /loss/FRt4/detection_loss‚2.9839696049690247 /loss/FBt2/detection_loss‚2.8930823604265847 /loss/FRt2/assignment_loss‚4.481398847368029 /loss/FBt3/detection_loss‚2.4365482444946585 1loss/FBt3/assignment_loss‚0.05787237395606308 1loss/FBt4/assignment_loss‚0.21036389860850963 /loss/FRt3/assignment_loss‚2.502419672693525 1loss/FBt2/assignment_loss‚0.37499674244059455 0loss/SRqqt2/detection_loss‚3.056311306628314 trainer/global_step‚3599 /loss/FRt4/assignment_loss‚4.555369472503662 2loss/SRqqt4/assignment_loss‚2.0726942469676337 /loss/FRt1/detection_loss‚1.9993991678411311 1loss/SRqqt3/detection_loss‚1.4473798895535404 /loss/FBt1/detection_loss‚1.2318789461278357 &loss/total_loss‚49.205544435334964ÞÉ7bÂÜ 2loss/SRqqt2/assignment_loss‚2.5502874038436194 0loss/FBt1/assignment_loss‚0.2186932456278301 0loss/SRqqt4/detection_loss‚2.830264567087094 2loss/SRqqt1/assignment_loss‚1.0813680938699028 /loss/FRt3/detection_loss‚2.7635677742106575 epoch‚2 % _timestamp‚1.7884315925590339e+09 _runtime‚3570.913984115 /loss/FRt4/detection_loss‚2.9839696049690247 /loss/FBt2/detection_loss‚2.8930823604265847 /loss/FRt2/assignment_loss‚4.481398847368029 /loss/FBt3/detection_loss‚2.4365482444946585 1loss/FBt3/assignment_loss‚0.05787237395606308 1loss/FBt4/assignment_loss‚0.21036389860850963 /loss/FRt3/assignment_loss‚2.502419672693525 1loss/FBt2/assignment_loss‚0.37499674244059455 0loss/SRqqt2/detection_loss‚3.056311306628314 trainer/global_step‚3599 /loss/FRt4/assignment_loss‚4.555369472503662 2loss/SRqqt4/assignment_loss‚2.0726942469676337 /loss/FRt1/detection_loss‚1.9993991678411311 1loss/SRqqt3/detection_loss‚1.4473798895535404 /loss/FBt1/detection_loss‚1.2318789461278357 &loss/total_loss‚49.205544435334964 _step‚293 0loss/FRt1/assignment_loss‚2.2612515276128597 2loss/SRqqt3/assignment_loss‚1.2654156561124406 .loss/FRt2/detection_loss‚2.622932129436069 1loss/SRqqt1/detection_loss‚1.5277014233849264 /loss/FBt4/detection_loss‚1.7803778395704601¥çFדÂ(:ˆ ö™åÔŒ®Ÿµ gpu.0.gpu‚40.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚125.833' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†äsåeÄ‚(:Ú ö™åÔ䬗¸ network.sent‚ 8.106823e+06 disk.drbd2080.out‚0 disk.sda.out‚ 130.73046875 disk.sda2.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚26608.76416015625 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.83984375 network.recv‚ 1.259209e+06 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 268.16796875 disk.drbd2266.out‚0 disk.rbd1.in‚0% disk./.usageGB‚276.50241470336914 disk.sda3.out‚ 130.73046875 disk.drbd1716.out‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 268.16796875 disk.drbd2265.in‚0 disk.dm-1.out‚ 130.73046875( disk.nvme0c0n1.out‚26608.76416015625) proc.memory.availableMB‚ 1540.88671875 cpu‚25.211282005921543) disk./.usagePercent‚18.55630905172188 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0# memory_percent‚90.5951738357544# proc.memory.rssMB‚ 2807.54296875) proc.memory.percent‚17.13588237762451 disk.dm-0.in‚ 268.16796875 disk.rbd0.out‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚26608.764160156257¼´“Å‚(:ˆ …šåԹ窵 gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚298.319' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.6638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g%[Æ‚(:Ð …šåÔ¾£û¶$ disk./.usageGB‚276.5021743774414 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚26808.66650390625 disk.sda3.in‚0 disk.sda.out‚ 131.2890625" proc.memory.rssMB‚ 2897.0234375 network.sent‚ 8.214913e+06# disk.dm-0.out‚26808.66650390625 disk.drbd1716.in‚0 disk.rbd0.out‚0 cpu‚25.214631440396516 network.recv‚ 1.265389e+06 disk.sda3.out‚ 131.2890625 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 1451.21875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0$ memory_percent‚91.14246368408203) proc.memory.percent‚17.68202781677246 proc.cpu.threads‚17) disk./.usagePercent‚18.55629292324722 disk.dm-0.in‚ 283.4609375 disk.drbd2080.out‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 283.4609375 disk.drbd2265.in‚0& disk.nvme0n1.out‚26808.66650390625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 283.4609375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd2265.out‚0 disk.dm-1.out‚ 131.28906256ŒlP’Ç‚(:‡ ”šåÔˆùú´ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚40.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^sO]È‚(:Ò ”šåÔ¿ûĸ disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 131.6171875 disk.sda.out‚ 131.6171875# proc.memory.rssMB‚ 2930.27734375" disk.dm-0.out‚26988.6943359375) proc.memory.availableMB‚ 1417.80859375 disk.nvme0n1.in‚ 293.91015625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375" disk.nvme0c0n1.in‚ 293.91015625' disk.nvme0c0n1.out‚26988.6943359375 disk.drbd2080.in‚0 cpu‚25.232814023739007) disk./.usagePercent‚18.55629292324722$ disk./.usageGB‚276.5021743774414# memory_percent‚91.3463830947876 disk.rbd0.out‚0% disk.nvme0n1.out‚26988.6943359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 293.91015625 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0 network.sent‚ 8.220214e+06 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.88499355316162 network.recv‚ 1.26907e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 131.6171875Ûð“6’É‚(:‡ £šåÔ“Ú°µ gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚302.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.5152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'G%ŠWÊ‚(:Ì £šåÔÿ”É· disk.nvme0n1.in‚ 304.44140625 disk.dm-1.out‚ 132.25390625" disk.dm-0.out‚27088.3623046875 disk.rbd0.in‚0( proc.memory.percent‚17.9595947265625 disk.drbd2266.out‚0 disk.sda.out‚ 132.25390625 disk.drbd2265.in‚0* disk./.usagePercent‚18.556296763360233 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0$ disk./.usageGB‚276.5022315979004 disk.sda2.out‚0 proc.memory.rssMB‚2942.5 disk.dm-1.in‚0 disk.sda.in‚0# memory_percent‚91.4163589477539& proc.memory.availableMB‚ 1406.34375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 304.44140625' disk.nvme0c0n1.out‚27088.3623046875 disk.sda3.in‚0 disk.drbd2265.out‚0 network.sent‚ 8.227085e+06 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 cpu‚25.249310829299116 proc.cpu.threads‚17 disk.sda3.out‚ 132.25390625 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.274278e+06% disk.nvme0n1.out‚27088.3623046875 disk.sda2.in‚0 disk.rbd1.out‚ 1.83984375 disk.dm-0.in‚ 304.44140625ÊŽu“Ë‚(:ˆ ²šåÔßáþ´ gpu.0.gpu‚96.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž Š®dÌ‚(:Ù ²šåÔ„܈· network.sent‚ 8.233448e+06( disk.nvme0c0n1.out‚27197.82568359375( proc.memory.availableMB‚ 1406.6953125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 314.83984375 disk.dm-0.in‚ 314.83984375# disk.dm-0.out‚27197.82568359375 disk.sda.out‚ 132.81640625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.nvme0n1.in‚ 314.83984375 disk.drbd2266.out‚0& disk.nvme0n1.out‚27197.82568359375$ memory_percent‚91.41421318054199 proc.cpu.threads‚17 disk.dm-1.out‚ 132.81640625 disk.drbd2266.in‚0 disk.sda3.out‚ 132.81640625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.960500717163086 network.recv‚ 1.278958e+06) disk./.usagePercent‚18.55629292324722 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚276.5021743774414 disk.sda.in‚0" proc.memory.rssMB‚ 2942.6484375 cpu‚25.266756037243894’]‚J2Í- _wandbruntime‚3650  _runtime‚3650âÆ;g¨&ÎãT BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚500531339496448 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7752934385664 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚28494363 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚968 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 143128400 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28216 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚3626 % _timestamp‚1.7884316724297268e+09 _runtime‚3650.785094598 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚500533693396992 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18282 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚484359002655744 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7737498848189 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492780759011328 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚65025119 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚431815972129792 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10757418556928 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353900544 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 143146682 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚38893108 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚492096501503933 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _step‚294 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚500531339496448 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353900544 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7750604779520 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27963 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606144 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚78103286 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚484359025986560 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 143128400 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67387611 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 127030272 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10757309175296 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021652 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693092 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚431815845099520 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7739826539025 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 143146682 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚78121563 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚140 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 109381632 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878631936 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690836 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚65025119 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚78103286 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27963 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚500533693396992 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚65025114 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚421058535924224 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚65025114 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492780734716928 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚78121563 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚421058553572864 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18277 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚492098852525585 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606144 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479245 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492780759011328 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18277BÏ8ýn*ÏèT I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18282 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492780734716928 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 25086444032 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7752934385664 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7750604779520 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚28494501 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚67387471 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚38893110 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚138 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18277 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚500531339496448 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7752934385664 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚28494363 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚968 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 143128400 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28216 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚3626 % _timestamp‚1.7884316724297268e+09 _runtime‚3650.785094598 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚500533693396992 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18282 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚484359002655744 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7737498848189 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492780759011328 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878631936 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚65025119 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚431815972129792 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10757418556928 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353900544 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 143146682 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚38893108 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚492096501503933 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _step‚294 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚500531339496448 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353900544 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7750604779520 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27963 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606144 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚78103286 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚484359025986560 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 143128400 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67387611 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 127030272 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10757309175296 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021652 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693092 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚431815845099520 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7739826539025 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 143146682 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚78121563 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚140 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 109381632 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878631936 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690836 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚65025119 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚78103286 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27963 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚500533693396992 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚65025114 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚421058535924224 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚65025114 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492780734716928 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚78121563 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚421058553572864 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18277 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚492098852525585 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606144 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479245 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492780759011328 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28216 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 trainer/global_step‚3626 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚492870327326720 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690836 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚484443952256000 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28216 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚65038317 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878631936 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 143167574 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492870351621120 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚78129257 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27963 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7751102796800 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚492870327326720 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚38902772 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 109381632 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606144 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456™tÿ; (ÒðR hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚421133830298112 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 143149292 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18277 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878631936 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27963 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚139 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚28498914 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚492184298577489 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 143167574 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 143149292 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚492181947555837 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚500623784024064 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67401686 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7753432402944 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚421133812649472 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 $ _timestamp‚1.788431673470577e+09 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚78129257 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚500621430123520 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18282 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021652 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7740322990673 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18277 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606144 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28499053 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7751102796800 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚492870351621120 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚65038312 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28216 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 trainer/global_step‚3626 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚492870327326720 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690836 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚484443952256000 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28216 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚65038317 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878631936 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 143167574 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492870351621120 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚78129257 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27963 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7751102796800 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚492870327326720 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚38902772 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 109381632 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606144 E?DeviceStatsMonitor.on_validation_batch_end/num_syn_ë‘kc_all_streams‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚65038312 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0§â ÉžÓ‚(:“ ÁšåÔŒ”Ûµ gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚302.864' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.572799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¼æXÔ‚(:Í ÁšåÔ‡Ùÿ· disk.dm-1.in‚0 disk.rbd1.out‚ 1.83984375 disk.dm-0.in‚ 325.109375 disk.sda1.in‚0 disk.drbd2080.in‚0) proc.memory.percent‚17.96085834503174 cpu‚25.28234300188103$ disk./.usageGB‚276.5026435852051& disk.nvme0n1.out‚27295.78662109375 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚27295.78662109375# disk.dm-0.out‚27295.78662109375 network.sent‚ 8.240487e+06 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 325.109375 disk.sda3.in‚0 disk.nvme0n1.in‚ 325.109375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 133.3046875 disk.sda3.out‚ 133.3046875 disk.drbd1716.in‚0 disk.sda.out‚ 133.3046875$ memory_percent‚91.41578674316406% proc.memory.availableMB‚ 1406.4375* disk./.usagePercent‚18.556324412173932 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2942.70703125 network.recv‚ 1.284616e+061©N2Õ- _wandbruntime‚3666  _runtime‚3666ÖÁïJxÖ£¥ 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4954843819141388 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3633721172809601 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.39043211936950684 ,jet/accuracy_3_of_4‚0.047375086694955826 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16764897108078003 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15475386381149292 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5073413252830505 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919964671134949 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926753640174866 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25718390941619873 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.022890469059348106 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2982342541217804 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17886601388454437 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.27258098125457764 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4027778208255768 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666666626930237 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2430555522441864 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920789003372192 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13608531653881073 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.35815539956092834 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20524679124355316 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13453246653079987 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.166691854596138 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01641705073416233 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18340301513671875 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969807267189026 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47621291875839233 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5615384578704834 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009352391585707664 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1641891896724701 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23991002142429352 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4772396981716156 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 % _timestamp‚1.7884316879442701e+09 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.34567901492118835 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18132716417312622 )loss/val_total_loss‚47.30969222483602 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010556815192103386 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5175594687461853 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.020071271806955338 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2198275625705719 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3851480185985565 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1882680207490921 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07995302975177765 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5731269121170044 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3978061378002167 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21222297847270966 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23685699701309204 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9885525107383728 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9236687421798706 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.39851313829421997 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965806603431702 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15069231390953064 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0587497241795063 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2150029093027115 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3453703224658966 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7validation_average_jet_accuracy‚0.37784984707832336 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33327242732048035 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24864184856414795 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19308125972747803 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.330000102519989 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11200738698244095 +"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_4_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.184199720621109 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875860810279846 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.33860090374946594 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 jet/accuracy_6_of_6‚0 jet/accuracy_5_of_7‚0 jet/accuracy_7_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2283630669116974 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.37859976291656494 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.32840919494628906 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.99033123254776 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,particle/sensitivity‚0.26560479402542114 _runtime‚3666.301238746 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.99025958776474 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6294863224029541 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20026862621307373 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3580845594406128 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/4FRt2SRqqt0FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.46111738681793213 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13198329508304596 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008712450973689556 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.37334972620010376 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49497124552726746 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9926303625106812 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.986709713935852 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.26881086826324463 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5396230816841125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.337545245885849 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13793103396892548 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5247043371200562 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5291282534599304 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.339407354593277 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3942011594772339 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3728039264678955 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12719298899173737 particle/accuracy_7_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14394931495189667 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4058506488800049 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17910727858543396 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02013806253671646 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5465054512023926 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_5‚1 *jet/accuracy_1_of_6‚0.9230769276618958 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810662567615509 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_6‚0 particle/accuracy_5_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08621269464492798 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3418983519077301 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4353448748588562 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.040265120565891266 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13634321093559265 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 epoch‚2 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.42105263471603394 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4616524577140808 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01344086043536663 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9914753437042236 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 jet/accuracy_5_of_5‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4100877344608307 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99812251329422 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993485152721405 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14865471422672272 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2080506831407547 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.276883065700531 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2765151560306549 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.020297538489103317 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19978435337543488 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.28419533371925354 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4194948673248291 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6886278986930847 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2959314286708832 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1632988154888153 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 ( Purity/4FRt2SRqqt*FBt/FRt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.32399407029151917 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13198329508304596 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011398476548492908 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2883104085922241 +jet/accuracy_2_of_2‚0.09355559945106506 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3274509310722351 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.019704435020685196 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3696759045124054 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4652777910232544 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906883835792542 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0026954179629683495 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941892623901367 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_7‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4058172106742859 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_1_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.33277270197868347 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5170455574989319 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33194443583488464 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03397245332598686 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2569023370742798 *jet/accuracy_1_of_3‚0.7965729832649231 particle/accuracy_2_of_7‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 5 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33664670586586 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18652093410491943 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9844507575035095 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3046131134033203 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5221419334411621 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3353370130062103 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1340879499912262 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9349881410598755 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28406447172164917 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3270089328289032 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013477866537868977 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19119884073734283 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_4‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3251633644104004 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.24189189076423645 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.20664575695991516 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9928749799728394 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20811082422733307 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3996817171573639 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3240070343017578 *jet/accuracy_2_of_4‚0.3427642285823822 +jet/accuracy_2_of_6‚0.42307692766189575 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17925839126110077 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_4‚0.8610489964485168 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 trainer/global_step‚3626 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5459460020065308 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.40164491534233093 ("Purity/*FRt2SRqq|‘B¸`t3FBt/event_purity‚0 _step‚296 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17644056677818298 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3998549282550812 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30215150117874146 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9902679920196533 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3257470726966858 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15378786623477936 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3418983519077301 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3879743814468384 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07967998832464218 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9830772876739502 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32623180747032166 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17259594798088074 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5823412537574768 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.37408459186553955 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2253585010766983 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36612024903297424 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15586736798286438 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1275002509355545 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.33470016717910767 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5061855912208557 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31198766827583313 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1133333370089531 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35255905985832214 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9933191537857056 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 %particle/f_score‚0.33214271068573 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5718749761581421 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20760232210159302 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004303477238863707 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.38907840847969055 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5175123810768127 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05252975970506668 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36175739765167236 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.446898877620697 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 )jet/accuracy_1_of_2‚0.668428897857666 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4099999964237213 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 +jet/accuracy_3_of_5‚0.05270097404718399 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.158558651804924 -jet/accuracy_4_of_4‚0.0011337868636474013 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1510416716337204 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14903315901756287 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01594662293791771 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989332914352417 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14819768071174622 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 (particle/accuracy‚0.8333693146705627 +particle/specificity‚0.9384108781814575 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986Ñü½‚’eר¥ @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4058172106742859 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_1_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.33277270197868347 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5170455574989319 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33194443583488464 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03397245332598686 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2569023370742798 *jet/accuracy_1_of_3‚0.7965729832649231 particle/accuracy_2_of_7‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 5 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33664670586586 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18652093410491943 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9844507575035095 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3046131134033203 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5221419334411621 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3353370130062103 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1340879499912262 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9349881410598755 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28406447172164917 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3270089328289032 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013477866537868977 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19119884073734283 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_4‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3251633644104004 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.24189189076423645 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.20664575695991516 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9928749799728394 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20811082422733307 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3996817171573639 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3240070343017578 *jet/accuracy_2_of_4‚0.3427642285823822 +jet/accuracy_2_of_6‚0.42307692766189575 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17925839126110077 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_4‚0.8610489964485168 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 trainer/global_step‚3626 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5459460020065308 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.40164491534233093 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚296 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17644056677818298 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3998549282550812 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30215150117874146 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9902679920196533 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3257470726966858 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15378786623477936 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3418983519077301 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3879743814468384 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07967998832464218 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9830772876739502 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32623180747032166 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17259594798088074 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5823412537574768 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.37408459186553955 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2253585010766983 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36612024903297424 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15586736798286438 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1275002509355545 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.33470016717910767 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5061855912208557 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31198766827583313 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1133333370089531 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35255905985832214 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9933191537857056 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 %particle/f_score‚0.33214271068573 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5718749761581421 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20760232210159302 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004303477238863707 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.38907840847969055 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5175123810768127 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05252975970506668 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36175739765167236 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.446898877620697 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 )jet/accuracy_1_of_2‚0.668428897857666 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4099999964237213 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 +jet/accuracy_3_of_5‚0.05270097404718399 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.158558651804924 -jet/accuracy_4_of_4‚0.0011337868636474013 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1510416716337204 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14903315901756287 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01594662293791771 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989332914352417 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14819768071174622 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 (particle/accuracy‚0.8333693146705627 +particle/specificity‚0.9384108781814575 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4954843819141388 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3633721172809601 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.39043211936950684 ,jet/accuracy_3_of_4‚0.047375086694955826 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16764897108078003 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15475386381149292 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5073413252830505 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919964671134949 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926753640174866 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25718390941619873 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.022890469059348106 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2982342541217804 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17886601388454437 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.27258098125457764 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4027778208255768 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666666626930237 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2430555522441864 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920789003372192 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13608531653881073 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.35815539956092834 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20524679124355316 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13453246653079987 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.166691854596138 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01641705073416233 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18340301513671875 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969807267189026 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.47621291875839233 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5615384578704834 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009352391585707664 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1641891896724701 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23991002142429352 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4772396981716156 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 % _timestamp‚1.7884316879442701e+09 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.34567901492118835 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18132716417312622 )loss/val_total_loss‚47.30969222483602 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010556815192103386 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5175594687461853 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.020071271806955338 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2198275625705719 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3851480185985565 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1882680207490921 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07995302975177765 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5731269121170044 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3978061378002167 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21222297847270966 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23685699701309204 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9885525107383728 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9236687421798706 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.39851313829421997 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965806603431702 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15069231390953064 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0587497241795063 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2150029093027115 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3453703224658966 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7validation_average_jet_accuracy‚0.37784984707832336 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33327242732048035 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24864184856414795 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19308125972747803 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.330000102519989 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11200738698244095 +"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_4_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.184199720621109 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875860810279846 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.33860090374946594 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 jet/accuracy_6_of_6‚0 jet/accuracy_5_of_7‚0 jet/accuracy_7_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2283630669116974 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.37859976291656494 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.32840919494628906 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.99033123254776 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,particle/sensitivity‚0.26560479402542114 _runtime‚3666.301238746 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.99025958776474 =&Purity/0FRt0SRqqt*FBt/¿”×:-event_proportion‚0.1191096231341362 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6294863224029541 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20026862621307373 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3580845594406128 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/4FRt2SRqqt0FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.46111738681793213 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13198329508304596 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008712450973689556 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.37334972620010376 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49497124552726746 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9926303625106812 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.986709713935852 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.26881086826324463 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5396230816841125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.337545245885849 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13793103396892548 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5247043371200562 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5291282534599304 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.339407354593277 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3942011594772339 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3728039264678955 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12719298899173737 particle/accuracy_7_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14394931495189667 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4058506488800049 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17910727858543396 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02013806253671646 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5465054512023926 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_5‚1 *jet/accuracy_1_of_6‚0.9230769276618958 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810662567615509 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_6‚0 particle/accuracy_5_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08621269464492798 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3418983519077301 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4353448748588562 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.040265120565891266 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13634321093559265 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 epoch‚2 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.42105263471603394 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4616524577140808 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01344086043536663 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9914753437042236 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 jet/accuracy_5_of_5‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4100877344608307 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99812251329422 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993485152721405 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14865471422672272 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2080506831407547 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.276883065700531 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2765151560306549 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.020297538489103317 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19978435337543488 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.28419533371925354 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4194948673248291 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6886278986930847 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2959314286708832 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1632988154888153 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 ( Purity/4FRt2SRqqt*FBt/FRt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.32399407029151917 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13198329508304596 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011398476548492908 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2883104085922241 +jet/accuracy_2_of_2‚0.09355559945106506 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3274509310722351 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.019704435020685196 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3696759045124054 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4652777910232544 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906883835792542 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0026954179629683495 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941892623901367 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_7‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1¨ÒÀ€’Ø‚(:‡ КåÔææ×µ gpu.0.gpu‚20.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4524 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëƒÔÉYÙ‚(:Î КåԇǷ) disk./.usagePercent‚18.70258228450801 disk.nvme0n1.in‚ 334.99609375 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 334.99609375 disk.sda1.out‚0 cpu‚25.288433297156477 network.recv‚ 1.290274e+06" disk.dm-0.out‚29544.8720703125 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚278.68199157714844 disk.drbd2080.in‚0 disk.sda.out‚ 133.65625 disk.drbd2265.in‚0 disk.dm-1.out‚ 133.65625 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.83984375( proc.memory.availableMB‚ 3913.8515625* proc.memory.percent‚21.399807929992676 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 133.65625$ memory_percent‚76.11174583435059 network.sent‚ 8.34815e+06 disk.drbd2080.out‚0% disk.nvme0n1.out‚29544.8720703125 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 334.99609375# proc.memory.rssMB‚ 3506.14453125' disk.nvme0c0n1.out‚29544.8720703125 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0|ðI3‘Ú‚(:† ßšåÔÑŒøµ gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ>UXÛ‚(:Í ßšåÔÛ¢Ž¸ disk.drbd2266.in‚0' disk.nvme0c0n1.out‚34199.3720703125 disk.dm-0.in‚ 347.796875 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 134.55859375 disk.rbd1.out‚ 1.92578125$ memory_percent‚72.70383834838867 cpu‚25.28450237302078 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 134.55859375' proc.memory.availableMB‚ 4472.203125* proc.memory.percent‚17.983198165893555 network.sent‚ 8.47857e+06 network.recv‚ 1.310118e+06) disk./.usagePercent‚18.70255437968678 disk.rbd0.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚278.6815757751465 disk.nvme0n1.in‚ 347.796875" disk.dm-0.out‚34199.3720703125 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0% disk.nvme0n1.out‚34199.3720703125" proc.memory.rssMB‚ 2946.3671875 disk.drbd2265.in‚0 disk.sda3.out‚ 134.55859375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 347.796875»Ø»©2Ü- _wandbruntime‚3701  _runtime‚3701Y#vÉÝà ' lr-AdamW/pg1‚0.00018022103199808172 ' lr-AdamW/pg2‚0.00018022103199808172 trainer/global_step‚3649 % _timestamp‚1.7884317226680954e+09 _runtime‚3701.023148192 _step‚297=O$ÉÎÞÈ % _timestamp‚1.7884317226680954e+09 _runtime‚3701.023148192 _step‚297 ' lr-AdamW/pg1‚0.00018022103199808172 ' lr-AdamW/pg2‚0.00018022103199808172 trainer/global_step‚3649©ϯ"ã2ß- _wandbruntime‚3701  _runtime‚3701KÈû„(àúO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76857344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚424585482295808 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504731178741248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67980297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚496927894994944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435415548090880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚496927921386496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 _runtime‚3701.024102142 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7793180105686 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28726840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚496927921386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7806389139968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10830065795072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78698074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78674968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65590133 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144265101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144288213 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504731178741248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚496927894994944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7790076636762 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65590133 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39253962 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884317226688492e+09 _step‚298 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67980805 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48722944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435415471233536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7806389139968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78674968Ó>×cÇáÿO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39253962 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884317226688492e+09 _step‚298 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67980805 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48722944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435415471233536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7806389139968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78674968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76857344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚424585482295808 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504731178741248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67980297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚496927894994944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435415548090880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚496927921386496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 _runtime‚3701.024102142 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7793180105686 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28726840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚496927921386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7806389139968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10830065795072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78698074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78674968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65590133 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144265101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144288213 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504731178741248 \HDeviceStatsMonitor.on_train_batch–@.> _start/allocated_bytes.large_pool.freed‚496927894994944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7790076636762 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65590133ª‹9ßï2â- _wandbruntime‚3701  _runtime‚3701^ó: 'ã„N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39264218 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144303475 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74746880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚497059415474688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67998320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28734573 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78696058 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7808501467136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 % _timestamp‚1.7884317235499995e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144326613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435530943928320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435530869181440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46612480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚424697865025536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7808501467136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39264215 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504867916941824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144303475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488524425990144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3701.905534092 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7805396060160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚497059389083136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28734105 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚496319740317106 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65607417 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144326613 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚496316611420006 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7805396060160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65607423 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7795288898994 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504867916941824 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78719190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚488524451418112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7792185429862P’5'ä‰N REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78719190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚488524451418112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7792185429862 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39264218 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144303475 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74746880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚497059415474688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67998320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28734573 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78696058 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7808501467136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 % _timestamp‚1.7884317235499995e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144326613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435530943928320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435530869181440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46612480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚424697865025536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7808501467136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39264215 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504867916941824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144303475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488524425990144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3701.905534092 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7805396060160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚497059389083136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28734105 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚496319740317106 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65607417 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144326613 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚496316611420006 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7805396060160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65607423 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7795288898994 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504867916941824 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249«pøµõ2å- _wandbruntime‚3701  _runtime‚3701ªžŠœ^æØ 1loss/SRqqt2/assignment_loss‚1.648281550925711 /loss/FRt3/detection_loss‚1.8422494473782454 /loss/FRt2/detection_loss‚2.6849809267691205 1loss/SRqqt2/detection_loss‚3.0401654217554177 1loss/SRqqt3/detection_loss‚1.1550322365947068 /loss/FBt1/detection_loss‚1.1631288914289326 epoch‚3 1loss/SRqqt1/detection_loss‚1.3276129064418503 /loss/FBt2/detection_loss‚2.4507801207629116 /loss/FBt4/detection_loss‚1.4703654612281494 0loss/FRt3/assignment_loss‚1.6501016746867787 0loss/FBt1/assignment_loss‚0.0966058528290955 0loss/FBt4/assignment_loss‚0.2158321662282103 /loss/FBt3/detection_loss‚2.7007534325122835 % _timestamp‚1.7884317235515895e+09 _step‚300 2loss/SRqqt1/assignment_loss‚1.3646921484841634 /loss/FRt1/detection_loss‚1.9275414857502078 trainer/global_step‚3649 _runtime‚3701.906561891 0loss/FRt2/assignment_loss‚5.1709140709468295 1loss/FBt2/assignment_loss‚0.48591511544178834 1loss/SRqqt4/detection_loss‚2.2856090991727767 0loss/FRt1/assignment_loss‚2.1935957097539713 1loss/SRqqt3/assignment_loss‚1.334611760410057 2loss/SRqqt4/assignment_loss‚1.6073471799853347 1loss/FBt3/assignment_loss‚0.10031272826090572 /loss/FRt4/detection_loss‚2.8414200169699533 /loss/FRt4/assignment_loss‚4.322858933040074 %loss/total_loss‚45.08070833775747nkÊõcçÝ /loss/FBt4/detection_loss‚1.4703654612281494 0loss/FRt3/assignment_loss‚1.6501016746867787 0loss/FBt1/assignment_loss‚0.0966058528290955 0loss/FBt4/assignment_loss‚0.2158321662282103 /loss/FBt3/detection_loss‚2.7007534325122835 % _timestamp‚1.7884317235515895e+09 _step‚300 2loss/SRqqt1/assignment_loss‚1.3646921484841634 /loss/FRt1/detection_loss‚1.9275414857502078 trainer/global_step‚3649 _runtime‚3701.906561891 0loss/FRt2/assignment_loss‚5.1709140709468295 1loss/FBt2/assignment_loss‚0.48591511544178834 1loss/SRqqt4/detection_loss‚2.2856090991727767 0loss/FRt1/assignment_loss‚2.1935957097539713 1loss/SRqqt3/assignment_loss‚1.334611760410057 2loss/SRqqt4/assignment_loss‚1.6073471799853347 1loss/FBt3/assignment_loss‚0.10031272826090572 /loss/FRt4/detection_loss‚2.8414200169699533 /loss/FRt4/assignment_loss‚4.322858933040074 %loss/total_loss‚45.08070833775747 1loss/SRqqt2/assignment_loss‚1.648281550925711 /loss/FRt3/detection_loss‚1.8422494473782454 /loss/FRt2/detection_loss‚2.6849809267691205 1loss/SRqqt2/detection_loss‚3.0401654217554177 1loss/SRqqt3/detection_loss‚1.1550322365947068 /loss/FBt1/detection_loss‚1.1631288914289326 epoch‚3 1loss/SRqqt1/detection_loss‚1.3276129064418503 /loss/FBt2/detection_loss‚2.4507801207629116¬Š”ëlžè‚(:“ îšåÔ‘•Ãµ gpu.0.gpu‚56.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚139.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.821800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÁœUé‚(:Ê îšåÔÿ¾Ó· disk.sda2.out‚0( disk.nvme0c0n1.out‚34299.68896484375 disk.dm-0.in‚ 355.65625 disk.sda.in‚0 network.recv‚ 1.327178e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 135.1484375# memory_percent‚72.6996898651123" proc.memory.rssMB‚ 2946.5078125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2080.in‚0) proc.memory.percent‚17.98405647277832* disk./.usagePercent‚18.702588428688834 disk.rbd0.out‚0 disk.dm-1.out‚ 135.1484375 disk.nvme0c0n1.in‚ 355.65625 disk.drbd2265.in‚0 cpu‚25.2834992393632 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.98828125( proc.memory.availableMB‚ 4472.8828125& disk.nvme0n1.out‚34299.68896484375 disk.rbd1.in‚0 disk.sda3.out‚ 135.1484375# disk.dm-0.out‚34299.68896484375 disk.sda3.in‚0 disk.drbd1716.in‚0 network.sent‚ 8.491007e+06 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0$ disk./.usageGB‚278.6820831298828 disk.rbd0.in‚0 disk.nvme0n1.in‚ 355.65625š£êžê‚(:“ ýšåÔ·„³¶ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚226.505' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.300999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãA_ë‚(:Ô ýšåÔ‹ÃÓ·( disk.nvme0c0n1.out‚34410.83154296875 disk.dm-1.in‚0 disk.drbd2266.in‚0# disk.dm-0.out‚34410.83154296875 disk.drbd1716.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚34410.83154296875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 136.08203125 disk.sda1.out‚0 disk.sda.out‚ 136.08203125 disk.drbd2266.out‚0 disk.dm-0.in‚ 367.3671875 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 367.3671875 disk.sda2.in‚0 disk.rbd1.out‚ 1.98828125 disk.drbd1716.out‚0' proc.memory.availableMB‚ 4471.984375* proc.memory.percent‚17.985105514526367 cpu‚25.28256395786972 network.recv‚ 1.351501e+06) disk./.usagePercent‚18.70279758684431 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0$ memory_percent‚72.70517349243164" proc.memory.rssMB‚ 2946.6796875 disk.dm-1.out‚ 136.08203125! disk.nvme0c0n1.in‚ 367.3671875 disk.drbd2265.in‚0 network.sent‚ 8.608901e+06$ disk./.usageGB‚278.6851997375488 disk.drbd2265.out‚0X¼¤õ“ì‚(:ˆ Œ›åÔì‰û´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚284.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡öF-[í‚(:Ð Œ›åÔ݈¸ network.recv‚ 1.35556e+06 disk.nvme0c0n1.in‚ 377.671875 disk.sda.in‚0 disk.sda.out‚ 136.50390625$ memory_percent‚72.69961833953857) disk./.usagePercent‚18.70280065893472 disk.rbd1.out‚ 2.0205078125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 377.671875 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 377.671875 disk.dm-1.out‚ 136.50390625' disk.nvme0c0n1.out‚34514.6083984375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 4472.89453125 cpu‚25.281562112517257% disk.nvme0n1.out‚34514.6083984375 disk.sda3.out‚ 136.50390625 disk.drbd2265.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2946.73046875 proc.cpu.threads‚17 network.sent‚ 8.614403e+06 disk.drbd1716.out‚0( proc.memory.percent‚17.9854154586792# disk./.usageGB‚278.685245513916 disk.sda2.out‚0 disk.drbd2266.out‚0" disk.dm-0.out‚34514.6083984375 disk.sda3.in‚0E2î- _wandbruntime‚3747  _runtime‚3747HŽaÏÉïà _step‚301 ' lr-AdamW/pg1‚0.00017925663614433082 ' lr-AdamW/pg2‚0.00017925663614433082 trainer/global_step‚3699 % _timestamp‚1.7884317688173647e+09 _runtime‚3747.172457529¶Q5ÎðÈ ' lr-AdamW/pg1‚0.00017925663614433082 ' lr-AdamW/pg2‚0.00017925663614433082 trainer/global_step‚3699 % _timestamp‚1.7884317688173647e+09 _runtime‚3747.172457529 _step‚301­Ýk,&2ñ- _wandbruntime‚3747  _runtime‚3747M*ßš-òúO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚511411513257472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441184296747008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 _runtime‚3747.173325827 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚511411513257472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚494864879049728 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29111013 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10978352296448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚511414645042688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503502599404544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚494864904477696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66454339 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503502625796096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503502625796096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68877106 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7908913852928 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79732820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79732820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7912019246592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66454333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878TO M(óÿO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚511414645042688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503502599404544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚494864904477696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66454339 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503502625796096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503502625796096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68877106 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7908913852928 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79732820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79732820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7912019246592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66454333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚511411513257472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441184296747008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 _runtime‚3747.173325827 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚511411513257472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚494864879049728 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29111013 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10978352296448®ñŸ§2ô- _wandbruntime‚3748  _runtime‚3748Ú¯ 'õ„N =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511545119659520 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _step‚303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146248665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441299732729856 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39776868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79777042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68895329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494994276050944 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66471617 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79753910 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7897638676070 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68894814 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511548251458048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146248665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79753910 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146225527 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66471617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7911026166784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441299651691520 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511548251458048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29117949 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430318436366336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494994301478912 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7914131573760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚3748.059466098 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146225527 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66471623 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39776865 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503634093492736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180eoÃÛÉö‰N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79777042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68895329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494994276050944 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66471617 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79753910 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7897638676070 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68894814 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511548251458048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146248665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79753910 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146225527 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66471617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7911026166784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441299651691520 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511548251458048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29117949 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430318436366336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494994301478912 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7914131573760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚3748.059466098 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146225527 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66471623 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39776865 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503634093492736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511545119659520 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _step‚303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146248665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441299732729856 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39776868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249¯…è>_2÷- _wandbruntime‚3748  _runtime‚3748|#&éXøÒ /loss/FBt4/assignment_loss‚0.216512783896178 1loss/SRqqt1/detection_loss‚1.0471160233412917 0loss/SRqqt4/detection_loss‚2.253774209664418 /loss/FRt1/assignment_loss‚1.886167640225929 1loss/FBt1/assignment_loss‚0.08241050312528017 epoch‚3 1loss/FBt2/assignment_loss‚0.45577432811001456 1loss/FBt3/assignment_loss‚0.24805099304885894 .loss/FRt2/detection_loss‚2.404566070612739 /loss/FRt2/assignment_loss‚4.447148266960593 /loss/FRt3/assignment_loss‚2.226109824618515 1loss/SRqqt3/detection_loss‚1.2185968253542394 _step‚304 /loss/FRt1/detection_loss‚1.8575686736587893 /loss/FBt1/detection_loss‚0.9947054556012154 % _timestamp‚1.7884317697054536e+09 0loss/FRt4/assignment_loss‚4.2970960934956866 2loss/SRqqt3/assignment_loss‚1.4848411813378335 2loss/SRqqt4/assignment_loss‚1.7023980021476746 1loss/SRqqt2/detection_loss‚2.8817185901460194 /loss/FBt2/detection_loss‚3.1092193172528195 -loss/FBt4/detection_loss‚1.71076749288477 &loss/total_loss‚45.490395299334665 trainer/global_step‚3699 2loss/SRqqt2/assignment_loss‚2.4082071653434207 /loss/FRt3/detection_loss‚2.0709192248023287 /loss/FRt4/detection_loss‚3.0774798343578973 _runtime‚3748.060413235 0loss/SRqqt1/assignment_loss‚1.36583015521204 .loss/FBt3/detection_loss‚2.043416644136111]Èk]ù× 1loss/FBt2/assignment_loss‚0.45577432811001456 1loss/FBt3/assignment_loss‚0.24805099304885894 .loss/FRt2/detection_loss‚2.404566070612739 /loss/FRt2/assignment_loss‚4.447148266960593 /loss/FRt3/assignment_loss‚2.226109824618515 1loss/SRqqt3/detection_loss‚1.2185968253542394 _step‚304 /loss/FRt1/detection_loss‚1.8575686736587893 /loss/FBt1/detection_loss‚0.9947054556012154 % _timestamp‚1.7884317697054536e+09 0loss/FRt4/assignment_loss‚4.2970960934956866 2loss/SRqqt3/assignment_loss‚1.4848411813378335 2loss/SRqqt4/assignment_loss‚1.7023980021476746 1loss/SRqqt2/detection_loss‚2.8817185901460194 /loss/FBt2/detection_loss‚3.1092193172528195 -loss/FBt4/detection_loss‚1.71076749288477 &loss/total_loss‚45.490395299334665 trainer/global_step‚3699 2loss/SRqqt2/assignment_loss‚2.4082071653434207 /loss/FRt3/detection_loss‚2.0709192248023287 /loss/FRt4/detection_loss‚3.0774798343578973 _runtime‚3748.060413235 0loss/SRqqt1/assignment_loss‚1.36583015521204 .loss/FBt3/detection_loss‚2.043416644136111 /loss/FBt4/assignment_loss‚0.216512783896178 1loss/SRqqt1/detection_loss‚1.0471160233412917 0loss/SRqqt4/detection_loss‚2.253774209664418 /loss/FRt1/assignment_loss‚1.886167640225929 1loss/FBt1/assignment_loss‚0.08241050312528017 epoch‚3°³*ˆú‚(:’ ››åÔÒ‘•µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚269.93' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.986000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾öéXû‚(:Í ››åÔ±¿†· disk.sda3.out‚ 137.21875 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 386.86328125 disk.dm-1.out‚ 137.21875# memory_percent‚72.7015495300293 proc.cpu.threads‚17 network.recv‚ 1.359974e+06 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 386.86328125 disk.drbd2080.out‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚34577.3193359375 disk.sda.out‚ 137.21875 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2946.73046875( proc.memory.percent‚17.9854154586792" disk.dm-0.out‚34577.3193359375 disk.drbd2080.in‚0 disk.drbd2266.in‚0* disk./.usagePercent‚18.681403549222917$ disk./.usageGB‚278.3664131164551 disk.sda1.in‚0% disk.nvme0n1.out‚34577.3193359375 disk.sda1.out‚0 network.sent‚ 8.620393e+06 disk.dm-0.in‚ 386.86328125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 4472.578125 cpu‚25.280658365389694 disk.rbd1.out‚ 2.0205078125 disk.sda3.in‚0 disk.sda2.in‚0™x“ü‚(:ˆ ª›åÔäü™µ gpu.0.gpu‚82.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚166.862' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¦{êký‚(:à ª›åÔŠˆ¯· disk.dm-0.in‚ 387.39453125 disk.sda3.out‚139.6875 disk.drbd2080.out‚0% disk.nvme0n1.out‚34602.1591796875 proc.cpu.threads‚17 disk.sda2.out‚0' disk.nvme0c0n1.out‚34602.1591796875 disk.sda.out‚139.6875 disk.drbd2265.in‚0 disk.sda2.in‚0$ memory_percent‚72.70221710205078! proc.memory.rssMB‚ 2946.828125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 387.39453125 disk.dm-1.in‚0.203125 disk.dm-1.out‚139.6875 disk.drbd2266.in‚0 network.sent‚ 8.728892e+06% disk./.usageGB‚278.36674880981445 disk.rbd1.in‚0 disk.rbd1.out‚ 2.08349609375* proc.memory.percent‚17.986011505126953 cpu‚25.279741441283544 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 network.recv‚ 1.366164e+06) disk./.usagePercent‚18.68142607788593 disk.sda.in‚0.203125& proc.memory.availableMB‚ 4472.46875" disk.dm-0.out‚34602.1591796875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 387.39453125JWžþ‚(:“ ¹›åÔô“™µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚128.511' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.702199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œp}Šgÿ‚(:Ü ¹›åÔï¹É·) disk./.usagePercent‚18.68265773013316 disk.rbd1.out‚ 2.08349609375 disk.rbd0.out‚0$ memory_percent‚72.70376682281494* proc.memory.percent‚17.986297607421875 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0" disk.nvme0c0n1.out‚ 34620.59375 disk.nvme0n1.in‚ 387.39453125 cpu‚25.278767130588328 network.recv‚ 1.371445e+06 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.nvme0n1.out‚ 34620.59375 disk.dm-0.in‚ 387.39453125 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 141.30859375 disk.rbd0.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 4472.21484375$ disk./.usageGB‚278.3851013183594 disk.rbd1.in‚0 disk.dm-0.out‚ 34620.59375 disk.sda3.out‚ 141.30859375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 proc.memory.rssMB‚2946.875 network.sent‚ 8.735704e+06" disk.nvme0c0n1.in‚ 387.39453125 disk.dm-1.out‚ 141.30859375 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.drbd1716.out‚0½nç½2€- _wandbruntime‚3793  _runtime‚3793µ4ƒÉà ' lr-AdamW/pg1‚0.00017827619956142698 ' lr-AdamW/pg2‚0.00017827619956142698 trainer/global_step‚3749 % _timestamp‚1.7884318154592159e+09 _runtime‚3793.814249928 _step‚305<“³ˆÎ‚È ' lr-AdamW/pg1‚0.00017827619956142698 ' lr-AdamW/pg2‚0.00017827619956142698 trainer/global_step‚3749 % _timestamp‚1.7884318154592159e+09 _runtime‚3793.814249928 _step‚305±É~÷2ƒ- _wandbruntime‚3793  _runtime‚3793[¶‘(„úO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29493083 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435826540574208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3749 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚518091847773696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11125670163968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _runtime‚3793.815183359 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11125723081216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚518091847773696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510077303814144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501334729110528 _step‚306 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚510077330205696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8014543959552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80790672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80813778 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148109205 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8017649353216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80813778 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40279262 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446952291789824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69772889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29493624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67318539 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8014543959552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67318533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40279265 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509335712239706 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8004086598102 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000983129178 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80790672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚518094979558912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8017649353216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚518094979558912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446952210738176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509338841136598 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501334754538496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚510077303814144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435826568708608 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148132317 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884318154599638e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚510077330205696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435826540574208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3749 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚518091847773696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11125670163968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _runtime‚3793.815183359 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11125723081216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚518091847773696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510077303814144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501334729110528 _step‚306 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚510077330205696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8014543959552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80790672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80813778 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148109205 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8017649353216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80813778 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40279262 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446952291789824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69772889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29493624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67318539 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8014543959552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67318533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40279265 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509335712239706 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8004086598102 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000983129178 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80790672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚518094979558912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8017649353216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚518094979558912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446952210738176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509338841136598 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501334754538496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚510077303814144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435826568708608 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.oÄŠËB•n_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148132317 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884318154599638e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚510077330205696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8019761680384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29501149 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80811762 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚510208797902336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67335823 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509467218034022 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884318163409767e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚3794.696733894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29500649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148170717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8016656273408 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚3749 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435938979572736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148147579 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510208824293888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447067564914688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501464126111744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148147579 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚518225454175744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚518228585974272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8003091922278 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8016656273408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚307 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8019761680384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69790667 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11128666380288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509470346931122 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚518228585974272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69790164 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447067645953024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80834894 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11128613476352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40289518 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510208824293888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40289515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435938951438336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚510208797902336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67335817 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148170717 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501464151539712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968L%'ˆ‰N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67335817 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148170717 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501464151539712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67335823 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8019761680384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29501149 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80811762 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚510208797902336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67335823 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509467218034022 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884318163409767e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚3794.696733894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29500649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148170717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8016656273408 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚3749 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435938979572736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148147579 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510208824293888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447067564914688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501464126111744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148147579 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚518225454175744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚518228585974272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8003091922278 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8016656273408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚307 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8019761680384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69790667 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11128666380288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509470346931122 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚518228585974272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69790164 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447067645953024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80834894 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11128613476352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40289518 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510208824293888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40289515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435938951438336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚510208797902336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864³€xì2‰- _wandbruntime‚3794  _runtime‚3794)œÐ¾YŠÓ 0loss/FRt1/assignment_loss‚2.3754802901866072 .loss/FRt1/detection_loss‚2.097190045577995 % _timestamp‚1.7884318163429227e+09 .loss/FRt3/assignment_loss‚2.24010602136453 0loss/FRt4/assignment_loss‚4.5720161733956175 /loss/FBt1/detection_loss‚0.8938720576741077 2loss/SRqqt3/assignment_loss‚1.3655920734901617 1loss/FBt3/assignment_loss‚0.14142997269518673 .loss/FRt3/detection_loss‚2.277901611135652 /loss/FBt4/detection_loss‚2.1105118200182913 epoch‚3 0loss/FBt2/assignment_loss‚0.5566325988103955 1loss/FBt4/assignment_loss‚0.10267295135768073 /loss/FRt4/detection_loss‚3.1882012835864364 1loss/SRqqt1/detection_loss‚1.1393901819219956 2loss/SRqqt1/assignment_loss‚1.3624533322854684 2loss/SRqqt2/assignment_loss‚1.9975857067367304 .loss/FRt2/detection_loss‚2.396747769628252 1loss/SRqqt3/detection_loss‚1.1758537145286072 0loss/SRqqt4/detection_loss‚2.196749211209161 _runtime‚3794.697876098 /loss/FRt2/assignment_loss‚4.294928891318185 2loss/FBt1/assignment_loss‚0.059368866717740464 /loss/FBt3/detection_loss‚2.6100627899169924 $loss/total_loss‚46.5635120138999 trainer/global_step‚3749 _step‚308 2loss/SRqqt4/assignment_loss‚2.0981710757528034 0loss/SRqqt2/detection_loss‚2.937735333390858 .loss/FBt2/detection_loss‚2.372858241200447"~À'^‹Ø 1loss/SRqqt3/detection_loss‚1.1758537145286072 0loss/SRqqt4/detection_loss‚2.196749211209161 _runtime‚3794.697876098 /loss/FRt2/assignment_loss‚4.294928891318185 2loss/FBt1/assignment_loss‚0.059368866717740464 /loss/FBt3/detection_loss‚2.6100627899169924 $loss/total_loss‚46.5635120138999 trainer/global_step‚3749 _step‚308 2loss/SRqqt4/assignment_loss‚2.0981710757528034 0loss/SRqqt2/detection_loss‚2.937735333390858 .loss/FBt2/detection_loss‚2.372858241200447 0loss/FRt1/assignment_loss‚2.3754802901866072 .loss/FRt1/detection_loss‚2.097190045577995 % _timestamp‚1.7884318163429227e+09 .loss/FRt3/assignment_loss‚2.24010602136453 0loss/FRt4/assignment_loss‚4.5720161733956175 /loss/FBt1/detection_loss‚0.8938720576741077 2loss/SRqqt3/assignment_loss‚1.3655920734901617 1loss/FBt3/assignment_loss‚0.14142997269518673 .loss/FRt3/detection_loss‚2.277901611135652 /loss/FBt4/detection_loss‚2.1105118200182913 epoch‚3 0loss/FBt2/assignment_loss‚0.5566325988103955 1loss/FBt4/assignment_loss‚0.10267295135768073 /loss/FRt4/detection_loss‚3.1882012835864364 1loss/SRqqt1/detection_loss‚1.1393901819219956 2loss/SRqqt1/assignment_loss‚1.3624533322854684 2loss/SRqqt2/assignment_loss‚1.9975857067367304 .loss/FRt2/detection_loss‚2.396747769628252´ƒ /žŒ‚(:“ È›åÔ€ŽÓµ gpu.0.gpu‚58.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.684999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ›/)u‚(:ê È›åÔíÃÌ· disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.in‚0.203125 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 cpu‚25.27786874477703* disk./.usagePercent‚18.683512027274922 disk.drbd1716.in‚0% disk.nvme0n1.out‚34623.2431640625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.146484375' disk.nvme0c0n1.out‚34623.2431640625 disk.sda3.out‚ 141.66015625 disk.sda.out‚ 141.66015625 disk.nvme0n1.in‚ 387.39453125" disk.dm-0.out‚34623.2431640625$ memory_percent‚72.70548343658447) proc.memory.availableMB‚ 4471.93359375 network.sent‚ 8.744233e+06 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 141.66015625 proc.memory.rssMB‚2946.875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 387.39453125* proc.memory.percent‚17.986297607421875 network.recv‚ 1.381928e+06% disk./.usageGB‚278.39783096313477 disk.dm-0.in‚ 387.39453125…{×PŽ‚(:’ ×›åÔ…»Èµ gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚294.438' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.88759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ£îo‚(:ä ×›åÔîÃÕ· disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.dm-0.in‚ 387.39453125" disk.dm-0.out‚34624.0673828125 disk.drbd1716.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚34624.0673828125" disk.nvme0c0n1.in‚ 387.39453125 disk.sda3.out‚ 142.3671875 disk.sda1.in‚0 disk.drbd2080.in‚0# memory_percent‚72.7064847946167) proc.memory.availableMB‚ 4471.76953125 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 142.3671875 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.sent‚ 8.852852e+06 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2946.875 network.recv‚ 1.38927e+06 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 387.39453125 disk.dm-1.out‚ 142.3671875 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚34624.0673828125 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.986297607421875 cpu‚25.27697660690019* disk./.usagePercent‚18.683672031983818$ disk./.usageGB‚278.4002151489258ÉÉ)“‚(:ˆ æ›åÔ…Êã´ gpu.0.gpu‚68.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚260.484' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽP “q‘‚(:æ æ›åÔ æø·% disk.nvme0n1.out‚34638.5087890625 disk.dm-0.in‚ 387.39453125 disk.sda3.out‚ 143.00390625 disk.sda1.out‚0 disk.drbd2080.in‚0 network.recv‚ 1.393525e+06) disk./.usagePercent‚18.68377085089204 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 4471.7421875 disk.drbd2266.out‚0 disk.sda1.in‚0$ memory_percent‚72.70665168762207 network.sent‚ 8.85869e+06 disk.sda.out‚ 143.00390625 cpu‚25.27602399898327 disk.sda2.in‚0" disk.dm-0.out‚34638.5087890625 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 387.39453125 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda.in‚0.203125 proc.memory.rssMB‚2946.875* proc.memory.percent‚17.986297607421875 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚278.4016876220703 disk.dm-1.out‚ 143.00390625 disk.rbd1.out‚ 2.146484375" disk.nvme0c0n1.in‚ 387.39453125' disk.nvme0c0n1.out‚34638.5087890625 disk.drbd1716.in‚0c0î’‚(:’ õ›åÔ¢éǵ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.304' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.86079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂDlªu“‚(:ê õ›åÔ¬Œ³· disk.sda.out‚ 143.515625 disk.sda1.in‚0$ memory_percent‚72.70665168762207 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 387.39453125 disk.sda3.in‚0.203125 disk.rbd0.out‚0! proc.memory.rssMB‚ 2946.921875 disk.dm-1.in‚0.203125# disk.dm-0.out‚34640.28369140625 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.recv‚ 1.398318e+06 disk.sda2.in‚0( disk.nvme0c0n1.out‚34640.28369140625 disk.sda3.out‚ 143.515625* proc.memory.percent‚17.986583709716797 network.sent‚ 8.865104e+06% disk./.usageGB‚278.40250396728516& disk.nvme0n1.out‚34640.28369140625 disk.rbd1.out‚ 2.146484375 disk.dm-0.in‚ 387.39453125 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0* disk./.usagePercent‚18.683825636504363 disk.sda2.out‚0 disk.dm-1.out‚ 143.515625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 4471.7421875 cpu‚25.275154215941445 proc.cpu.threads‚17 disk.nvme0n1.in‚ 387.39453125 disk.rbd1.in‚0 disk.drbd1716.out‚0\·²‰2”- _wandbruntime‚3840  _runtime‚3840Zrí›Ç•Á _runtime‚3840.212211157 _step‚309 & lr-AdamW/pg1‚0.0001772799265708845 & lr-AdamW/pg2‚0.0001772799265708845 trainer/global_step‚3799 % _timestamp‚1.7884318618571913e+09Ôú­ÓÌ–Æ trainer/global_step‚3799 % _timestamp‚1.7884318618571913e+09 _runtime‚3840.212211157 _step‚309 & lr-AdamW/pg1‚0.0001772799265708845 & lr-AdamW/pg2‚0.0001772799265708845µ(À+Ã2—- _wandbruntime‚3840  _runtime‚3840·š˜úO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441447083780608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚524772182289920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚516652034615296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40791912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452720292087808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68182739 R>DeviceStatsMonitor.on_train_batch_start/active_bytes9H ".all.freed‚524772182289920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8123279459840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452720373139456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68182739 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81848524 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8106436375386 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚516652008223744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚524775314075136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441447111915008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11273261224448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68182733 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8109539844310 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 trainer/global_step‚3799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150054369 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70666582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8106436375386 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚516652008223744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚524775314075136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441447111915008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11273261224448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68182733 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8109539844310 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 trainer/global_step‚3799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150054369 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70666582 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441447083780608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚524772182289920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚516652034615296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40791912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452720292087808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68182739 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚524772182289920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8123279459840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452720373139456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68182739 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81848524 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8125391787008 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441559522779136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81892746 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚524905788691968 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68200023 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8108545168486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚311 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150092769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516783528703488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11276215116800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507933976172544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70684535 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68200017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚ 3841.09605537 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8122286380032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81892746 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81869614 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8125391787008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516783502311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68200017 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441559494644736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8111648637618 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8122286380032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚524908920490496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70684028 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11276162212864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150092769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29882367 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29881863 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452835656857600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441559494644736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8111648637618 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8122286380032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚524908920490496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70684028 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11276162212864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150092769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29882367 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29881863 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452835656857600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytesª?súò.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8125391787008 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441559522779136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81892746 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚524905788691968 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68200023 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8108545168486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚311 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150092769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516783528703488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11276215116800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507933976172544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70684535 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68200017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚ 3841.09605537·þmµ2- _wandbruntime‚3841  _runtime‚3841¸ÃéTžÎ 0loss/FBt2/assignment_loss‚0.5998915627133101 1loss/SRqqt2/detection_loss‚3.2766582338433516 /loss/FBt1/detection_loss‚0.8316389424405222 /loss/FBt4/detection_loss‚1.4998378898416247 .loss/FRt1/assignment_loss‚2.40593360228972 /loss/FRt3/detection_loss‚1.9213489079084554 %loss/total_loss‚47.84149757409233 _step‚312 1loss/SRqqt4/detection_loss‚2.2982447250136016 trainer/global_step‚3799 _runtime‚ 3841.09718444 2loss/SRqqt2/assignment_loss‚2.3549220373756006 2loss/SRqqt4/assignment_loss‚2.2620910812040855 0loss/FBt1/assignment_loss‚0.2362779603798287 1loss/SRqqt1/detection_loss‚1.5113051903702446 .loss/FBt2/detection_loss‚2.942075252532959 0loss/FBt4/assignment_loss‚0.3083537132181963 /loss/FRt1/detection_loss‚1.9713751485176159 /loss/FRt2/detection_loss‚2.5543226525187492 .loss/FRt4/assignment_loss‚4.89738815691736 2loss/SRqqt1/assignment_loss‚1.0873588406307213 2loss/SRqqt3/assignment_loss‚1.1896290295561285 0loss/FBt3/assignment_loss‚0.2572621185914599 .loss/FRt4/detection_loss‚2.705101364188724 epoch‚3 0loss/SRqqt3/detection_loss‚1.434962221960488 .loss/FBt3/detection_loss‚2.026127526164055 % _timestamp‚1.7884318627422118e+09 .loss/FRt2/assignment_loss‚4.99656068533659 0loss/FRt3/assignment_loss‚2.2728307305789386³•a9YŸÓ /loss/FRt1/detection_loss‚1.9713751485176159 /loss/FRt2/detection_loss‚2.5543226525187492 .loss/FRt4/assignment_loss‚4.89738815691736 2loss/SRqqt1/assignment_loss‚1.0873588406307213 2loss/SRqqt3/assignment_loss‚1.1896290295561285 0loss/FBt3/assignment_loss‚0.2572621185914599 .loss/FRt4/detection_loss‚2.705101364188724 epoch‚3 0loss/SRqqt3/detection_loss‚1.434962221960488 .loss/FBt3/detection_loss‚2.026127526164055 % _timestamp‚1.7884318627422118e+09 .loss/FRt2/assignment_loss‚4.99656068533659 0loss/FRt3/assignment_loss‚2.2728307305789386 0loss/FBt2/assignment_loss‚0.5998915627133101 1loss/SRqqt2/detection_loss‚3.2766582338433516 /loss/FBt1/detection_loss‚0.8316389424405222 /loss/FBt4/detection_loss‚1.4998378898416247 .loss/FRt1/assignment_loss‚2.40593360228972 /loss/FRt3/detection_loss‚1.9213489079084554 %loss/total_loss‚47.84149757409233 _step‚312 1loss/SRqqt4/detection_loss‚2.2982447250136016 trainer/global_step‚3799 _runtime‚ 3841.09718444 2loss/SRqqt2/assignment_loss‚2.3549220373756006 2loss/SRqqt4/assignment_loss‚2.2620910812040855 0loss/FBt1/assignment_loss‚0.2362779603798287 1loss/SRqqt1/detection_loss‚1.5113051903702446 .loss/FBt2/detection_loss‚2.942075252532959 0loss/FBt4/assignment_loss‚0.3083537132181963¸ ¼a“ ‚(:ˆ „œåÔ ‹ø´ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚229.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úãq¡‚(:æ „œåÔ՚ж network.recv‚ 1.401729e+06 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 387.39453125 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚34642.90234375# proc.memory.rssMB‚ 2947.07421875$ disk./.usageGB‚278.4037895202637 disk.drbd1716.in‚0" disk.nvme0c0n1.in‚ 387.39453125) proc.memory.availableMB‚ 4471.44921875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.out‚ 144.1796875 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 2.146484375* proc.memory.percent‚17.987513542175293 cpu‚25.274294453068748 disk.drbd2265.out‚0 disk.dm-1.out‚ 144.1796875 disk.drbd2266.out‚0 network.sent‚ 8.870412e+06 disk.sda3.out‚ 144.1796875 disk.rbd1.in‚0 disk.dm-0.out‚34642.90234375* disk./.usagePercent‚18.683911911043403 disk.drbd2080.in‚0# disk.nvme0n1.out‚34642.90234375 disk.drbd2266.in‚0$ memory_percent‚72.70843982696533 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 387.39453125U„%$“¢‚(:ˆ “œåÔ¡ŠÃµ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚128.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œÿ¨Yj£‚(:ß “œåÔòÑÛ· disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 34643.453125 disk.dm-0.in‚ 387.39453125 disk.drbd2080.out‚0 disk.sda.out‚ 144.7734375" disk.nvme0c0n1.in‚ 387.39453125 disk.dm-0.out‚ 34643.453125# proc.memory.rssMB‚ 2947.17578125* proc.memory.percent‚17.988133430480957 cpu‚25.273416099320013 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 network.sent‚ 8.97903e+06 disk.nvme0n1.in‚ 387.39453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 2.146484375$ memory_percent‚72.70750999450684( proc.memory.availableMB‚ 4471.6015625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚0.203125! disk.nvme0n1.out‚ 34643.453125 network.recv‚ 1.409994e+06% disk./.usageGB‚278.40446853637695 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 144.7734375 disk.drbd2266.out‚0 disk.sda3.out‚ 144.7734375 proc.cpu.threads‚17* disk./.usagePercent‚18.683957480384496";õ “¤‚(:ˆ ¢œåÔ÷Á©µ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚283.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÎ7÷w¥‚(:ì ¢œåÔÀܳ· disk.sda1.out‚0) proc.memory.percent‚17.98872947692871 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚34645.1806640625" disk.dm-0.out‚34645.1806640625 disk.sda1.in‚0" proc.memory.rssMB‚ 2947.2734375 cpu‚25.272499402326712 network.recv‚ 1.414025e+06) disk./.usagePercent‚18.68401021793655 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 387.39453125 disk.dm-0.in‚ 387.39453125) proc.memory.availableMB‚ 4471.54296875$ disk./.usageGB‚278.4052543640137 disk.drbd2265.out‚0 network.sent‚ 8.984853e+06 disk.drbd2080.in‚0 disk.sda.out‚ 145.26171875 disk.rbd0.in‚0% disk.nvme0n1.out‚34645.1806640625 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 145.26171875 disk.rbd1.out‚ 2.146484375 disk.sda3.in‚0.203125 disk.sda2.out‚0$ memory_percent‚72.70786762237549 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 387.39453125 disk.drbd2266.in‚0 disk.sda.in‚0.203125 disk.rbd1.in‚0 disk.sda3.out‚ 145.26171875Ü÷<È2¦- _wandbruntime‚3886  _runtime‚3886©<͖ȧ ' lr-AdamW/pg1‚0.00017626802479450143 ' lr-AdamW/pg2‚0.00017626802479450143 trainer/global_step‚3849 $ _timestamp‚1.788431908348436e+09 _runtime‚3886.703490325 _step‚313 t8—ꬂ ' lr-AdamW/pg1‚0.00017626802479450143 ' lr-AdamW/pg2‚0.00017626802479450143 trainer/global_step‚3849 $ _timestamp‚1.788431908348436e+09 _runtime‚3886.703490325 _step‚313¹9[±s2©- _wandbruntime‚3886  _runtime‚3886“ü ÿ'ªùO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531452516806144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523226739024896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8225804172800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447067626987008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8228909566464 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82906376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447067655121408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11420793161216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522489447750614 _runtime‚3886.704778073 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 $ _timestamp‚1.788431908349443e+09 _step‚314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69046939 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523226739024896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30257143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151976421 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522486318853722 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚514274429232128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11420848175616 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69046933 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523226712633344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8228909566464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30256605 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69046933 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41304565 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82929482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71561167 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8225804172800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69046939 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8211889621594 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531455648591360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 trainer/global_step‚3849 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964½’Él¼«þO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151976421 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522486318853722 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚514274429232128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11420848175616 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69046933 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523226712633344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8228909566464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30256605 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69046933 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41304565 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82929482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71561167 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8225804172800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69046939 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8211889621594 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531455648591360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 trainer/global_step‚3849 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531452516806144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523226739024896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8225804172800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447067626987008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8228909566464 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 31053936`Ý°H64 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82906376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447067655121408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11420793161216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522489447750614 _runtime‚3886.704778073 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 $ _timestamp‚1.788431908349443e+09 _step‚314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69046939 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523226739024896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30257143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878ºuÉz¾2¬- _wandbruntime‚3887  _runtime‚3887™ìèª '­„N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522617824648038 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚514403851661312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82927466 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚458603858138112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151991683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69064217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151991683 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41314815 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚447180065985536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11423737151488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚3849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531589255006720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8217101883826 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11423792152576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531586123208192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71578921 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458603775002624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69064223 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8231021893632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82927466 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41314818 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚315 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30264629 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152014821 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152014821 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82950598 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82950598 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522620953545138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚447180037851136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3887.591658656 ¿#Ö'®‰N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522617824648038 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚514403851661312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82927466 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚458603858138112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151991683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69064217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151991683 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41314815 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚447180065985536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11423737151488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚3849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531589255006720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8217101883826 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11423792152576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531586123208192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71578921 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458603775002624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69064223 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8231021893632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82927466 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41314818 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚315 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30264629 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152014821 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152014821 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82950598 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82950598 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522620953545138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚447180037851136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3887.591658656»¾ãô2¯- _wandbruntime‚3887  _runtime‚3887¹WÆW°Ñ .loss/FRt2/detection_loss‚2.824907220326937 0loss/SRqqt2/detection_loss‚2.656700779612248 /loss/FBt2/detection_loss‚2.5964118377728895 trainer/global_step‚3849 1loss/FBt1/assignment_loss‚0.06009738239248656 /loss/FRt4/detection_loss‚3.0702989407948085 /loss/FBt1/detection_loss‚1.0633356102063494 0loss/FRt1/assignment_loss‚2.3994168785153605 2loss/SRqqt2/assignment_loss‚1.9181499481201172 /loss/FRt1/detection_loss‚1.9992329726109699 0loss/SRqqt1/detection_loss‚1.338794896348578 &loss/total_loss‚46.999839429735225 $ _timestamp‚1.788431909237759e+09 .loss/FRt2/assignment_loss‚4.32219183139312 2loss/SRqqt3/assignment_loss‚1.1892447931215513 0loss/FBt2/assignment_loss‚0.5422234481031238 0loss/SRqqt4/detection_loss‚2.784801791543546 /loss/FBt4/detection_loss‚2.0223101310688874 _runtime‚3887.592696603 /loss/FRt3/assignment_loss‚2.194867196471192 /loss/FRt3/detection_loss‚2.2044302422639936 _step‚316 /loss/FRt4/assignment_loss‚4.384603663853237 0loss/FBt3/assignment_loss‚0.4172228118924977 1loss/FBt4/assignment_loss‚0.30343128627564825 .loss/FBt3/detection_loss‚2.663886469602585 epoch‚3 1loss/SRqqt1/assignment_loss‚1.299798872505055 2loss/SRqqt4/assignment_loss‚1.4914943867403527 1loss/SRqqt3/detection_loss‚1.2519860381996908È™ ò\±Ö /loss/FRt1/detection_loss‚1.9992329726109699 0loss/SRqqt1/detection_loss‚1.338794896348578 &loss/total_loss‚46.999839429735225 $ _timestamp‚1.788431909237759e+09 .loss/FRt2/assignment_loss‚4.32219183139312 2loss/SRqqt3/assignment_loss‚1.1892447931215513 0loss/FBt2/assignment_loss‚0.5422234481031238 0loss/SRqqt4/detection_loss‚2.784801791543546 /loss/FBt4/detection_loss‚2.0223101310688874 _runtime‚3887.592696603 /loss/FRt3/assignment_loss‚2.194867196471192 /loss/FRt3/detection_loss‚2.2044302422639936 _step‚316 /loss/FRt4/assignment_loss‚4.384603663853237 0loss/FBt3/assignment_loss‚0.4172228118924977 1loss/FBt4/assignment_loss‚0.30343128627564825 .loss/FBt3/detection_loss‚2.663886469602585 epoch‚3 1loss/SRqqt1/assignment_loss‚1.299798872505055 2loss/SRqqt4/assignment_loss‚1.4914943867403527 1loss/SRqqt3/detection_loss‚1.2519860381996908 .loss/FRt2/detection_loss‚2.824907220326937 0loss/SRqqt2/detection_loss‚2.656700779612248 /loss/FBt2/detection_loss‚2.5964118377728895 trainer/global_step‚3849 1loss/FBt1/assignment_loss‚0.06009738239248656 /loss/FRt4/detection_loss‚3.0702989407948085 /loss/FBt1/detection_loss‚1.0633356102063494 0loss/FRt1/assignment_loss‚2.3994168785153605 2loss/SRqqt2/assignment_loss‚1.9181499481201172¼ê‡#{“²‚(:ˆ ±œåÔÌÿ¤µ gpu.0.gpu‚52.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚304.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.8344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ʼ(w³‚(:ì ±œåÔ‡¢· disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.drbd1716.in‚0 network.recv‚ 1.418716e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 387.80859375% disk.nvme0n1.out‚34645.8486328125' disk.nvme0c0n1.out‚34645.8486328125* disk./.usagePercent‚18.684063467503673$ disk./.usageGB‚278.4060478210449 disk.dm-1.out‚ 145.90234375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.sda3.in‚0.203125" proc.memory.rssMB‚ 2947.3046875 cpu‚25.27165112356043 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 387.80859375 disk.sda3.out‚ 145.90234375 proc.cpu.threads‚17 disk.sda.out‚ 145.90234375 disk.drbd2265.in‚0" disk.dm-0.out‚34645.8486328125 disk.sda1.out‚0* proc.memory.percent‚17.988920211791992 network.sent‚ 8.991184e+06 disk.rbd1.out‚ 2.146484375 disk.sda1.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 978.47265625$ memory_percent‚94.02787685394287ˆ»±´‚(:’ ÀœåÔû’µ gpu.0.gpu‚69.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.94' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.587999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07¸šêsµ‚(:è ÀœåÔ§ÙÞ¸ disk.rbd0.in‚0 network.sent‚ 9.099163e+06 network.recv‚ 1.425002e+06 disk.sda3.out‚ 146.28515625 disk.sda1.out‚0 disk.drbd2080.out‚0* disk./.usagePercent‚18.684103916694085 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0% proc.memory.availableMB‚ 978.90625 disk.nvme0n1.in‚ 387.80859375$ disk.nvme0n1.out‚34648.494140625 disk.sda2.out‚0 disk.dm-0.in‚ 387.80859375 disk.rbd0.out‚0 disk.drbd2266.in‚0$ memory_percent‚94.02523040771484* proc.memory.percent‚17.989516258239746 disk.drbd1716.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 2.146484375# proc.memory.rssMB‚ 2947.40234375 disk.sda.in‚0.203125 disk.dm-1.out‚ 146.28515625 cpu‚25.270787812286137 disk.dm-1.in‚0.203125& disk.nvme0c0n1.out‚34648.494140625 disk.rbd1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚278.4066505432129" disk.nvme0c0n1.in‚ 387.80859375! disk.dm-0.out‚34648.494140625 disk.sda3.in‚0.203125 disk.sda.out‚ 146.28515625 disk.drbd2265.out‚0ôÛž“¶‚(:ˆ ÏœåÔÌ¢µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚267.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^›ßZ·‚(:Ï ÏœåÔµ©†¸ cpu‚25.26989665659849 disk.sda1.out‚0 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 387.80859375! disk.dm-0.out‚34649.205078125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 387.80859375 network.recv‚ 1.428952e+06 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.146484375 disk.dm-0.in‚ 387.80859375 disk.sda3.out‚147 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.991328239440918 disk.sda.out‚147 disk.sda2.out‚0( proc.memory.availableMB‚ 978.46484375 proc.cpu.threads‚17) disk./.usagePercent‚18.68413463759819 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.out‚0$ memory_percent‚94.02792453765869 network.sent‚ 9.104908e+06 disk.dm-1.out‚147& disk.nvme0c0n1.out‚34649.205078125$ disk.nvme0n1.out‚34649.205078125 disk.drbd2266.out‚0 disk.drbd2080.in‚0% disk./.usageGB‚278.40710830688477 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2947.69921875¾Û$†2¸- _wandbruntime‚3932  _runtime‚3932yàœ`ɹà ' lr-AdamW/pg2‚0.00017524070511109124 trainer/global_step‚3899 % _timestamp‚1.7884319546128051e+09 _runtime‚3932.967847013 _step‚317 ' lr-AdamW/pg1‚0.00017524070511109124/&ÎºÈ ' lr-AdamW/pg2‚0.00017524070511109124 trainer/global_step‚3899 % _timestamp‚1.7884319546128051e+09 _runtime‚3932.967847013 _step‚317 ' lr-AdamW/pg1‚0.00017524070511109124½ʬ½Ì2»- _wandbruntime‚3932  _runtime‚3932«×Î> ¼úO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153875361 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30636872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529801443434496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8317342867802 % _timestamp‚1.7884319546135354e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538132851322368 TGDeviceStatd˜~øtsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69911139 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41817215 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464256460514816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚318 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8334539673088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚464256381560320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452688198327808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529801417042944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69911139 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8331434279424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520744304720896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83987334 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72454603 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452688170193408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72454084 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520744279292928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8334539673088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69911133 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83964228 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11568211366912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8331434279424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41817212 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11568262187008 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83987334 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30637388 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚529064751057622 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538132851322368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8320446336726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3932.968811552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30636872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529801443434496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8317342867802 % _timestamp‚1.7884319546135354e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538132851322368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69911139 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41817215 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464256460514816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚318 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8334539673088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚464256381560320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452688198327808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529801417042944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69911139 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8331434279424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520744304720896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83987334 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72454603 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452688170193408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72454084 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520744279292928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8334539673088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69911133 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83964228 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11568211366912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8331434279424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41817212 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11568262187008 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83987334 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30637388 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚529064751057622 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538132851322368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8320446336726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3932.968811552¾†>v2¾- _wandbruntime‚3933  _runtime‚3933ûpC '¿ƒN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529932937522688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153913735 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83985318 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153936873 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520873676294144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8319451660902 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72471743 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69928417 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529932911131136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚538269589522944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84008450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41827465 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30644278 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11571199840768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚464371727994368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8322555130034 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69928423 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚319 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529193127955046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚464371809032704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538269589522944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8333546593280 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529932937522688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30644780 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538266457724416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69928423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41827468 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520873701722112 trainer/global_step‚3899 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153936873 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520873676294144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8319451660902 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72471743 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69928417 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529932911131136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚538269589522944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84008450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41827465 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30644278 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11571199840768 REDeviceStatsMonitor.on_train_batch_end,6ðÅi/active_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚464371727994368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8322555130034 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69928423 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚319 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529193127955046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚464371809032704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538269589522944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8333546593280 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529932937522688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30644780 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302¿’‹&Ç2Á- _wandbruntime‚3933  _runtime‚3933I£YÂÓ 0loss/FBt2/assignment_loss‚0.5396146312910227 /loss/FBt4/detection_loss‚1.5743308131183897 2loss/SRqqt3/assignment_loss‚1.4193407114595176 1loss/FBt1/assignment_loss‚0.33926509018734397 /loss/FRt3/detection_loss‚2.2831653033693633 /loss/FRt1/assignment_loss‚2.012003700281012 /loss/FBt3/detection_loss‚2.8300328453381858 %loss/total_loss‚47.43011997208889 /loss/FRt1/detection_loss‚1.9384703564232793 1loss/SRqqt1/detection_loss‚1.3661941451951862 trainer/global_step‚3899 0loss/FRt3/assignment_loss‚2.7607365610698857 2loss/SRqqt1/assignment_loss‚1.0097231849329547 2loss/SRqqt2/assignment_loss‚1.6214123283823332 .loss/FBt2/detection_loss‚2.490306569979741 _step‚320 /loss/FRt2/detection_loss‚2.3719677527745566 /loss/FRt4/detection_loss‚2.9533248609966702 .loss/FBt1/detection_loss‚1.073351086055239 % _timestamp‚1.7884319554979117e+09 _runtime‚3933.852895239 /loss/FRt2/assignment_loss‚4.505537146177048 1loss/SRqqt4/assignment_loss‚1.802654274064919 0loss/FBt3/assignment_loss‚0.6152652764570803 1loss/SRqqt2/detection_loss‚2.6635269711414975 0loss/SRqqt4/detection_loss‚2.371546084510869 epoch‚3 /loss/FRt4/assignment_loss‚4.831528024541007 0loss/FBt4/assignment_loss‚0.4385681074449126 1loss/SRqqt3/detection_loss‚1.6182541468968759áÖÍÆ^ÃØ %loss/total_loss‚47.43011997208889 /loss/FRt1/detection_loss‚1.9384703564232793 1loss/SRqqt1/detection_loss‚1.3661941451951862 trainer/global_step‚3899 0loss/FRt3/assignment_loss‚2.7607365610698857 2loss/SRqqt1/assignment_loss‚1.0097231849329547 2loss/SRqqt2/assignment_loss‚1.6214123283823332 .loss/FBt2/detection_loss‚2.490306569979741 _step‚320 /loss/FRt2/detection_loss‚2.3719677527745566 /loss/FRt4/detection_loss‚2.9533248609966702 .loss/FBt1/detection_loss‚1.073351086055239 % _timestamp‚1.7884319554979117e+09 _runtime‚3933.852895239 /loss/FRt2/assignment_loss‚4.505537146177048 1loss/SRqqt4/assignment_loss‚1.802654274064919 0loss/FBt3/assignment_loss‚0.6152652764570803 1loss/SRqqt2/detection_loss‚2.6635269711414975 0loss/SRqqt4/detection_loss‚2.371546084510869 epoch‚3 /loss/FRt4/assignment_loss‚4.831528024541007 0loss/FBt4/assignment_loss‚0.4385681074449126 1loss/SRqqt3/detection_loss‚1.6182541468968759 0loss/FBt2/assignment_loss‚0.5396146312910227 /loss/FBt4/detection_loss‚1.5743308131183897 2loss/SRqqt3/assignment_loss‚1.4193407114595176 1loss/FBt1/assignment_loss‚0.33926509018734397 /loss/FRt3/detection_loss‚2.2831653033693633 /loss/FRt1/assignment_loss‚2.012003700281012 /loss/FBt3/detection_loss‚2.8300328453381858À©|îžÄ‚(:“ ÞœåÔ‰ ›µ gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚136.472' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.294400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A´P­tÅ‚(:é ÞœåÔ¿Œê¶ disk.sda.out‚ 147.578125 disk.drbd2266.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 978.05078125 proc.cpu.threads‚17 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd1.out‚ 2.146484375( disk.nvme0c0n1.out‚34652.41162109375# disk.dm-0.out‚34652.41162109375 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 387.80859375 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda3.out‚ 147.578125 network.recv‚ 1.432822e+06 disk.dm-0.in‚ 387.80859375 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚34652.41162109375* proc.memory.percent‚17.993450164794922* disk./.usagePercent‚18.683914215111212% disk./.usageGB‚278.40382385253906 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0$ memory_percent‚94.03045177459717! proc.memory.rssMB‚ 2948.046875 disk.rbd0.out‚0 disk.dm-1.out‚ 147.578125" disk.nvme0c0n1.in‚ 387.80859375 cpu‚25.26901907245347 network.sent‚ 9.109996e+06Õ²¾žÆ‚(:“ íœåÔ’µî´ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n¬TÔyÇ‚(:î íœåÔ£Áθ# disk./.usageGB‚278.404239654541 disk.rbd1.out‚ 2.146484375 disk.sda3.out‚ 148.11328125 disk.sda.in‚0.203125( proc.memory.availableMB‚ 978.29296875 cpu‚25.268171359903185 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd2265.out‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.sda.out‚ 148.11328125$ memory_percent‚94.02897357940674 network.sent‚ 9.219111e+06 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 network.recv‚ 1.440438e+06 disk.nvme0n1.in‚ 387.80859375 disk.dm-1.out‚ 148.11328125 disk.rbd1.in‚0 disk.dm-0.in‚ 387.80859375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2265.in‚0* disk./.usagePercent‚18.683942119932443" disk.nvme0c0n1.in‚ 387.80859375 disk.drbd1716.in‚0* proc.memory.percent‚17.993450164794922& disk.nvme0n1.out‚34653.61474609375( disk.nvme0c0n1.out‚34653.61474609375# disk.dm-0.out‚34653.61474609375! proc.memory.rssMB‚ 2948.046875ûˆ/¢‘È‚(:† üœåÔ¤òá´ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚300.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä÷äozÉ‚(:ï üœåÔñÛ´·$ disk./.usageGB‚278.4048080444336( disk.nvme0c0n1.out‚34657.30419921875$ memory_percent‚94.02892589569092 network.recv‚ 1.443925e+06 disk.drbd2266.out‚0# disk.dm-0.out‚34657.30419921875 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 148.51953125 proc.cpu.threads‚17& disk.nvme0n1.out‚34657.30419921875" disk.nvme0c0n1.in‚ 387.80859375( proc.memory.availableMB‚ 978.30078125! proc.memory.rssMB‚ 2948.046875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.146484375 disk.dm-0.in‚ 387.80859375* proc.memory.percent‚17.993450164794922 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125* disk./.usagePercent‚18.683980265055045 disk.rbd0.in‚0 disk.nvme0n1.in‚ 387.80859375 disk.sda3.out‚ 148.51953125 network.sent‚ 9.224401e+06 disk.dm-1.out‚ 148.51953125 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 cpu‚25.267306753050857g í2Ê- _wandbruntime‚3979  _runtime‚3979’™£ÇËÁ % _timestamp‚1.7884320010234838e+09 _runtime‚3979.378524062 _step‚321 & lr-AdamW/pg1‚0.0001741981816125362 & lr-AdamW/pg2‚0.0001741981816125362 trainer/global_step‚3949f’FõÌÌÆ & lr-AdamW/pg2‚0.0001741981816125362 trainer/global_step‚3949 % _timestamp‚1.7884320010234838e+09 _runtime‚3979.378524062 _step‚321 & lr-AdamW/pg1‚0.0001741981816125362ÁœØg÷2Í- _wandbruntime‚3979  _runtime‚39790¶ñÝ(ÎúO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85022080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8437064386048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85045186 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70775333 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8425899582934 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚536376121452544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85022080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70775333 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42329862 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _runtime‚3979.379437174 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚536376121452544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚535636925467738 trainer/global_step‚3949 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚535640054364630 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8440169779712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚527214129353728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚544816317623808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚544813185838592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚536376147844096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42329865 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚527214154781696 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70775339 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85045186 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚544813185838592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚536376147844096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470024289388544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8422796114010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚544816317623808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73348831 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458308741534208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 Jý$ÏÿO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85022080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8437064386048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85045186 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70775333 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8425899582934 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚536376121452544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85022080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70775333 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42329862 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _runtime‚3979.379437174 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚536376121452544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚535636925467738 trainer/global_step‚3949 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚535640054364630 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8440169779712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚527214129353728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚544816317623808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚544813185838592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚536376147844096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42329865 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚527214154781696 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70775339 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85045186 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚544813185838592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚536376147844096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470024289388544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8422796114010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚544816317623808 QDDeviceStatsMonitor.on_train_batch_start/inactive_sd÷§òplit.all.allocated‚73348831 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458308741534208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8442282106880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _step‚323 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470139645487616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚536507641932288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70792617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70792623 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155835787 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85043170 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚535768431262054 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73366139 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _runtime‚3980.264101279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85043170 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11718574127616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚536507641932288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155858925 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85066302 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31026024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155858925 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11718521223680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚536507615540736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155835787 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8439176699904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚506 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31026527 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚535771560159154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42340115 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚527343551782912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73366645 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42340118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458421152398336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884320019086926e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 trainer/global_step‚3949 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ž°/e'Ò‰N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 trainer/global_step‚3949 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8442282106880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _step‚323 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470139645487616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚536507641932288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70792617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70792623 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155835787 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85043170 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚535768431262054 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73366139 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _runtime‚3980.264101279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85043170 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11718574127616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚536507641932288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155858925 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85066302 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31026024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155858925 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11718521223680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚536507615540736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155835787 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8439176699904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚506 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31026527 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚535771560159154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42340115 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚527343551782912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73366645 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42340118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458421152398336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884320019086926e+09æAà2Ó- _wandbruntime‚3980  _runtime‚3980VA(ZÔÔ 0loss/SRqqt4/assignment_loss‚2.39182870449691 /loss/FRt1/detection_loss‚1.6323234295472502 1loss/SRqqt3/detection_loss‚1.4853758262076848 .loss/FBt3/detection_loss‚2.471051275730133 .loss/FBt2/detection_loss‚3.233961137858304 % _timestamp‚1.7884320019102368e+09 _runtime‚3980.265186947 0loss/FRt1/assignment_loss‚2.4565392561256885 1loss/SRqqt1/detection_loss‚1.1659560010601313 /loss/FBt1/detection_loss‚1.1735136894465878 /loss/FRt2/assignment_loss‚4.680890575699184 /loss/FRt4/assignment_loss‚3.939575183391571 1loss/FBt2/assignment_loss‚0.40773531752612063 1loss/SRqqt4/detection_loss‚2.4709128758002974 /loss/FBt4/detection_loss‚1.5443457668026288 _step‚324 0loss/FRt3/assignment_loss‚2.9184146034411893 2loss/SRqqt2/assignment_loss‚2.1165664259231454 2loss/SRqqt3/assignment_loss‚1.3747042605495998 1loss/FBt3/assignment_loss‚0.22451421690626602 0loss/FBt4/assignment_loss‚0.2662792047931968 /loss/FRt2/detection_loss‚2.4899503005587538 /loss/FRt4/detection_loss‚2.7137903749942778 2loss/SRqqt1/assignment_loss‚1.1474393524038486 1loss/FBt1/assignment_loss‚0.06883033823466343 .loss/FRt3/detection_loss‚2.371013097656079 0loss/SRqqt2/detection_loss‚2.319870578520226 %loss/total_loss‚47.06538179367374 epoch‚3 trainer/global_step‚3949Ü£Õ_ÕÙ _runtime‚3980.265186947 0loss/FRt1/assignment_loss‚2.4565392561256885 1loss/SRqqt1/detection_loss‚1.1659560010601313 /loss/FBt1/detection_loss‚1.1735136894465878 /loss/FRt2/assignment_loss‚4.680890575699184 /loss/FRt4/assignment_loss‚3.939575183391571 1loss/FBt2/assignment_loss‚0.40773531752612063 1loss/SRqqt4/detection_loss‚2.4709128758002974 /loss/FBt4/detection_loss‚1.5443457668026288 _step‚324 0loss/FRt3/assignment_loss‚2.9184146034411893 2loss/SRqqt2/assignment_loss‚2.1165664259231454 2loss/SRqqt3/assignment_loss‚1.3747042605495998 1loss/FBt3/assignment_loss‚0.22451421690626602 0loss/FBt4/assignment_loss‚0.2662792047931968 /loss/FRt2/detection_loss‚2.4899503005587538 /loss/FRt4/detection_loss‚2.7137903749942778 2loss/SRqqt1/assignment_loss‚1.1474393524038486 1loss/FBt1/assignment_loss‚0.06883033823466343 .loss/FRt3/detection_loss‚2.371013097656079 0loss/SRqqt2/detection_loss‚2.319870578520226 %loss/total_loss‚47.06538179367374 epoch‚3 trainer/global_step‚3949 0loss/SRqqt4/assignment_loss‚2.39182870449691 /loss/FRt1/detection_loss‚1.6323234295472502 1loss/SRqqt3/detection_loss‚1.4853758262076848 .loss/FBt3/detection_loss‚2.471051275730133 .loss/FBt2/detection_loss‚3.233961137858304 % _timestamp‚1.7884320019102368e+09ÄäõK“Ö‚(:ˆ ‹åÔÈ‹«¶ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚173.846' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]S{ׂ(:ð ‹åÔûÕî· network.recv‚ 1.448086e+06 disk.sda1.out‚0( proc.memory.availableMB‚ 977.70703125 proc.cpu.threads‚17 disk.drbd2266.out‚0 network.sent‚ 9.230375e+06& disk.nvme0n1.out‚34658.11279296875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.146484375 disk.drbd2080.in‚0 disk.sda.out‚ 149.13671875* proc.memory.percent‚17.995858192443848 cpu‚25.266488896873714% disk./.usageGB‚278.40502548217773 disk.nvme0n1.in‚ 387.80859375 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.dm-0.in‚ 387.80859375 disk.sda1.in‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2948.44140625# disk.dm-0.out‚34658.11279296875 disk.sda3.out‚ 149.13671875( disk./.usagePercent‚18.6839948574845 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 149.13671875 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚34658.11279296875 disk.sda3.in‚0.203125 disk.drbd1716.in‚0$ memory_percent‚94.03254985809326—n„Ǟ؂(:“ šåԶ˩µ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.121' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.624199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"üœvÙ‚(:ë šåÔï¦þ¶ disk.dm-1.in‚0.203125 disk.drbd2266.in‚0$ disk./.usageGB‚278.4058647155762 disk.drbd2080.out‚0 disk.dm-1.out‚ 149.58203125# disk.dm-0.out‚34659.83447265625 disk.sda3.in‚0.203125% proc.memory.availableMB‚ 977.71875* proc.memory.percent‚17.997026443481445 cpu‚25.26563275224925 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 387.80859375 disk.sda3.out‚ 149.58203125 proc.cpu.threads‚17 network.recv‚ 1.454895e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.drbd2265.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚34659.83447265625( disk.nvme0c0n1.out‚34659.83447265625 disk.sda1.in‚0" proc.memory.rssMB‚ 2948.6328125 disk.sda.out‚ 149.58203125 disk.nvme0n1.in‚ 387.80859375 disk.rbd1.out‚ 2.146484375$ memory_percent‚94.03247833251953 network.sent‚ 9.338866e+06) disk./.usagePercent‚18.68405117914203 disk.drbd2080.in‚0¦v»’Ú‚(:’ ©åÔßñ²µ gpu.0.gpu‚68.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.766' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.75320000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W"…{Û‚(:ð ©åÔìÅ­·* disk./.usagePercent‚18.684018410177647 disk.dm-0.in‚ 387.80859375# disk.dm-0.out‚34660.56884765625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 387.80859375 disk.dm-1.out‚ 150.14453125 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0$ memory_percent‚94.03274059295654 cpu‚25.264828425654663 network.sent‚ 9.345214e+06 network.recv‚ 1.459723e+06 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd2080.in‚0* proc.memory.percent‚17.997026443481445 disk.sda2.out‚0( disk.nvme0c0n1.out‚34660.56884765625 disk.sda.out‚ 150.14453125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375" disk.nvme0c0n1.in‚ 387.80859375 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2948.6328125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 150.14453125 disk.sda.in‚0.203125$ disk./.usageGB‚278.4053764343262& disk.nvme0n1.out‚34660.56884765625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 977.67578125 ð€€2Ü- _wandbruntime‚4025  _runtime‚4025•~ÆÈÝ _runtime‚4025.960103122 _step‚325 ' lr-AdamW/pg1‚0.00017314067155917086 ' lr-AdamW/pg2‚0.00017314067155917086 trainer/global_step‚3999 $ _timestamp‚1.788432047605089e+09©Ÿï ÍÞÇ $ _timestamp‚1.788432047605089e+09 _runtime‚4025.960103122 _step‚325 ' lr-AdamW/pg1‚0.00017314067155917086 ' lr-AdamW/pg2‚0.00017314067155917086 trainer/global_step‚3999Å~‡Ê2ß- _wandbruntime‚4025  _runtime‚4025Æå7'àúO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542215357671638 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475791966947328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86079932 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31401689 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚4025.960964617 _step‚326 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 % _timestamp‚1.7884320476057787e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.curreng´lXÙt‚ 3131785216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8545799886336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157742577 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚551493520354816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚542950852253696 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11862710341120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 % _timestamp‚1.7884320476057787e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8545799886336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157742577 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚551493520354816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚542950852253696 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11862710341120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542215357671638 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302ÆJ Dð2â- _wandbruntime‚4026  _runtime‚4026qWTÛ 'ã„N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475907415243776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464041695604736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533813376415744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551630258555392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86124154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8533461622450 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8547912213504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11865666735104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31408725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157757839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71656817 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71656817 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74261980 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157757839 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86101022 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8530358153318 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551627126756864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31409212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475907334205440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464041667470336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543082319950336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚487 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74261490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543082346341888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8547912213504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543082319950336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42852765 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71656823 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8544806806528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157780977 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542346863466162 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884320484902623e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8544806806528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71656823 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542343734569062 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3999 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚533813401843712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86124154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551630258555392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86101022 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11865719639040 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _step‚327 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74261980 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157757839 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86101022 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8530358153318 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551627126756864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31409212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475907334205440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464041667470336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543082319950336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚487 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74261490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543082346341888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segmenŽülLts.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8547912213504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543082319950336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42852765 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71656823 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8544806806528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157780977 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542346863466162 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884320484902623e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8544806806528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71656823 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542343734569062 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3999 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚533813401843712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86124154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551630258555392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86101022 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11865719639040 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _step‚327 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75139006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636806075350 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549525556663296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558176986656256 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43355165 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚548787532081754 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558173854871040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43355162 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159664629 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31783844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159641517 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72503739 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87160890 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87137784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558173854871040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚540153829475328 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87137784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚549525530271744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚549525530271744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12010189744640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12010244759040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚548790660978646 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8633702606426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469549827947008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72503733 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚558176986656256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8648324599296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481560072706048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8651429992960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72503739 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75139538 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚540153854903296 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 % _timestamp‚1.7884320938789635e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12010244759040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚548790660978646 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8633702606426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469549827947008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72503733 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚558176986656256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8648324599296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481560072706048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8651429992960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72503739 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75139538 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚540153854903296 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 % _timestamp‚1.7884320938789635e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75139006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636806075350 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549525556663296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558176986656256 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43355165 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚548787532081754 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558173854871040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43355162 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159664629 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31783844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159641517 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72503739 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87160890 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87137784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558173854871040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚540153829475328 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87137784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚549525530271744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚549525530271744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12010189744640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159703029 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚558310593071616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8650436913152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87158874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚469662210676736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558310593071616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43365415 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8653542320128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549657024359936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8650436913152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481675421888512 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72521017 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87182006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43365418 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549657024359936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481675338753024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚540283226476544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8635811399526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚549657050751488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 $ _timestamp‚1.788432094761391e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75157365 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75156843 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚548922166773170 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8638914868658 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87182006 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚331 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72521023 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12013183077376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159679891 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚540283251904512 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 trainer/global_step‚4049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72521023 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31791428 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159703029 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚548919037876070 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488ÛO&'öˆN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558310593071616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43365415 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8653542320128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549657024359936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8650436913152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481675421888512 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72521017 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87182006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43365418 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549657024359936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481675338753024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚540283226476544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8635811399526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚549657050751488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 $ _timestamp‚1.788432094761391e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75157365 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75156843 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚548922166773170 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8638914868658 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87182006 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚331 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72521023 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12013183077376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159679891 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚540283251904512 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 trainer/global_step‚4049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72521023 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31791428 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159703029 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚548919037876070 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12013128076288 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159703029 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚558310593071616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8650436913152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87158874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚469662210676736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878Ë[ä2÷- _wandbruntime‚4073  _runtime‚4073ï/*ZøÔ 2loss/SRqqt3/assignment_loss‚1.2210050061748354 1loss/SRqqt4/assignment_loss‚2.251343947190505 0loss/SRqqt4/detection_loss‚3.533053838289701 &loss/total_loss‚48.721528881492574 _runtime‚4073.118130744 1loss/FBt1/assignment_loss‚0.07994532395819816 1loss/SRqqt1/detection_loss‚1.4464950636029243 /loss/FRt4/assignment_loss‚4.954710517610822 2loss/SRqqt2/assignment_loss‚1.9235682551349913 /loss/FRt4/detection_loss‚3.1727674858910695 1loss/SRqqt3/detection_loss‚1.2397148467830776 trainer/global_step‚4049 $ _timestamp‚1.788432094763183e+09 /loss/FRt1/detection_loss‚2.2426326312827616 0loss/FRt3/assignment_loss‚2.3349776854590765 /loss/FRt2/detection_loss‚2.5863961862481157 /loss/FRt3/detection_loss‚1.8289892478358178 .loss/FBt1/detection_loss‚1.386396317000224 .loss/FBt3/detection_loss‚2.436941646039486 /loss/FRt1/assignment_loss‚2.241883740110217 1loss/FBt4/assignment_loss‚0.07586069311846365 epoch‚3 _step‚332 1loss/FBt2/assignment_loss‚0.23156798077532306 0loss/SRqqt2/detection_loss‚2.749436842543738 .loss/FRt2/assignment_loss‚4.61730576598126 1loss/FBt3/assignment_loss‚0.49402350458694855 /loss/FBt2/detection_loss‚2.8807029815820546 /loss/FBt4/detection_loss‚1.7045898843142722 2loss/SRqqt1/assignment_loss‚1.0872194899786864û×Þ•_ùÙ /loss/FRt4/assignment_loss‚4.954710517610822 2loss/SRqqt2/assignment_loss‚1.9235682551349913 /loss/FRt4/detection_loss‚3.1727674858910695 1loss/SRqqt3/detection_loss‚1.2397148467830776 trainer/global_step‚4049 $ _timestamp‚1.788432094763183e+09 /loss/FRt1/detection_loss‚2.2426326312827616 0loss/FRt3/assignment_loss‚2.3349776854590765 /loss/FRt2/detection_loss‚2.5863961862481157 /loss/FRt3/detection_loss‚1.8289892478358178 .loss/FBt1/detection_loss‚1.386396317000224 .loss/FBt3/detection_loss‚2.436941646039486 /loss/FRt1/assignment_loss‚2.241883740110217 1loss/FBt4/assignment_loss‚0.07586069311846365 epoch‚3 _step‚332 1loss/FBt2/assignment_loss‚0.23156798077532306 0loss/SRqqt2/detection_loss‚2.749436842543738 .loss/FRt2/assignment_loss‚4.61730576598126 1loss/FBt3/assignment_loss‚0.49402350458694855 /loss/FBt2/detection_loss‚2.8807029815820546 /loss/FBt4/detection_loss‚1.7045898843142722 2loss/SRqqt1/assignment_loss‚1.0872194899786864 2loss/SRqqt3/assignment_loss‚1.2210050061748354 1loss/SRqqt4/assignment_loss‚2.251343947190505 0loss/SRqqt4/detection_loss‚3.533053838289701 &loss/total_loss‚48.721528881492574 _runtime‚4073.118130744 1loss/FBt1/assignment_loss‚0.07994532395819816 1loss/SRqqt1/detection_loss‚1.4464950636029243Ì Mžú‚(:“ ååÔÇƶ gpu.0.gpu‚94.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.448' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.889599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘[¦jWû‚(:Ì ååԚภdisk.dm-0.out‚34666.44921875 disk.drbd2080.out‚0 network.sent‚ 9.471441e+06 network.recv‚ 1.479266e+06# disk.nvme0n1.out‚34666.44921875 disk.rbd1.out‚ 2.146484375 disk.sda.out‚152.5 disk.rbd0.in‚0& proc.memory.availableMB‚ 977.421875 disk.nvme0n1.in‚ 387.80859375 disk.drbd2265.out‚0# memory_percent‚94.0342903137207 disk.drbd2080.in‚0 disk.dm-1.out‚152.5 disk.drbd2266.in‚0% disk.nvme0c0n1.out‚34666.44921875 disk.dm-0.in‚ 387.80859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0$ disk./.usageGB‚278.4112243652344 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda.in‚0.203125) proc.memory.percent‚17.99788475036621 cpu‚25.26167403647273) disk./.usagePercent‚18.68441086972764" disk.nvme0c0n1.in‚ 387.80859375 disk.sda3.out‚152.5 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2948.7734375 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0.203125 proc.cpu.threads‚17ìùc±œü‚(:‘ ôåÔÈà͵ gpu.0.gpu‚69.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚296.432' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.28640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æî…Ópý‚(:å ôåÔüô–· disk.drbd2266.in‚0) proc.memory.percent‚17.99788475036621 proc.cpu.threads‚17 disk.dm-0.in‚ 387.80859375 disk.sda3.in‚0.203125$ memory_percent‚94.03433799743652 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 152.91796875$ disk./.usageGB‚278.4118881225586& disk.nvme0c0n1.out‚34670.615234375 disk.sda3.out‚ 152.91796875 disk.dm-1.in‚0.203125 network.recv‚ 1.485643e+06* disk./.usagePercent‚18.684455415038595 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚34670.615234375' proc.memory.availableMB‚ 977.4140625 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 387.80859375 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0 disk.sda.out‚ 152.91796875 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.drbd2080.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2948.7734375 cpu‚25.2608600092592 network.sent‚ 9.57952e+06! disk.dm-0.out‚34670.615234375 disk.sda1.in‚0 disk.drbd2265.out‚0{šÄü“þ‚(:ˆ ƒžåÔšˆ©¶ gpu.0.gpu‚69.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½š-Òsÿ‚(:è ƒžåÔ¶Ïû· network.sent‚ 9.585458e+06 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚34671.470703125 disk.sda1.in‚0 disk.nvme0n1.in‚ 387.80859375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 153.54296875 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda3.out‚ 153.54296875 disk.drbd2080.in‚0$ memory_percent‚94.03440952301025$ disk./.usageGB‚278.4124946594238$ disk.nvme0n1.out‚34671.470703125" disk.nvme0c0n1.in‚ 387.80859375" proc.memory.rssMB‚ 2948.7734375) proc.memory.percent‚17.99788475036621 disk.drbd2265.in‚0! disk.dm-0.out‚34671.470703125( proc.memory.availableMB‚ 977.40234375 cpu‚25.26008516509355* disk./.usagePercent‚18.684496120236542 disk.sda.out‚ 153.54296875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 387.80859375 disk.drbd1716.in‚0 network.recv‚ 1.490163e+06 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17r¬.2€- _wandbruntime‚4118  _runtime‚4118_\:PÇÁ % _timestamp‚1.7884321403413844e+09 _runtime‚4118.696519346 _step‚333 & lr-AdamW/pg1‚0.0001709815763992975 & lr-AdamW/pg2‚0.0001709815763992975 trainer/global_step‚4099‹æL Ì‚Æ & lr-AdamW/pg1‚0.0001709815763992975 & lr-AdamW/pg2‚0.0001709815763992975 trainer/global_step‚4099 % _timestamp‚1.7884321403413844e+09 _runtime‚4118.696519346 _step‚333Ík5d2ƒ- _wandbruntime‚4118  _runtime‚4118‚niìq„úO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚564857321172480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73367933 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43867815 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32161688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚556100234681344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚564854189387264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚556100234681344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8757060099584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _runtime‚4118.697407531 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32162205 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12157181044736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73367939 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88218742 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚334 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8757060099584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73367939 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88195636 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884321403421514e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactiv³TÖ~e_split_bytes.small_pool.allocated‚12157236059136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88195636 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8753954705920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76029500 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8753954705920 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475170343019008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8753954705920 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475170343019008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚564854189387264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚556100234681344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8757060099584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _runtime‚4118.697407531 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32162205 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12157181044736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73367939 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88218742 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚334 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8757060099584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73367939 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88195636 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884321403421514e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112ÎJùþ©2†- _wandbruntime‚4119  _runtime‚4119¯¥Š '‡„N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚564987795789312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76047355 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475282782017536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161625081 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555497470080178 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546753076537344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8756067019776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43878068 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88216726 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 _runtime‚4119.577534477 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚564990927587840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚556231755161088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73385223 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76047874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88216726 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 _runtime‚4119.577534477 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚564990927587840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚556231755161088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73385223 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76047874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76047355 cODeviceStatsMonitor.on_train_batch_end/inacti-ëÉú ve_split_bytes.large_pool.allocated‚475282782017536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161625081 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555497470080178 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546753076537344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8756067019776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43878068 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚571534523903488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561938138695770 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚571534523903488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76924472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884321868296463e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚571537655688704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493095310288896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89253488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553093529596928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74232133 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562674965482496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163508733 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74232133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32544007 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553093555024896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8844609098842ʘMu(—ÿO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12304479077888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44380462 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89253488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89276594 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89276594 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚571534523903488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561938138695770 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚571534523903488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76924472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884321868296463e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚571537655688704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493095310288896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89253488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553093529596928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74232133 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562674965482496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163508733 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74232133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32544007 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553093555024896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8844609098842 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561941267592662 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562674965482496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚338 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8862690206208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74232139ÒîÛÏ2˜- _wandbruntime‚4166  _runtime‚4166¢(I×<™„N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163523995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493210675670016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8846717891942 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553222952026112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12307378580480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562806433179136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553222926598144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493210758805504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89274578 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8849821361074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8864802533376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44390718 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74249423 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚571668130305536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32551022 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884321877101576e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12307433581568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562806459570688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74249417 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚571671262104064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8861697126400 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163547133 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89297710 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚562072773387186 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚562069644490086 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89297710 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚4149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44390715 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163547133 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562806459570688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76942245 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76941737 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4166.065924605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89274578 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562806433179136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚480903297089536 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74249417z„'š‰N UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8861697126400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562806433179136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚480903297089536 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74249417 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163523995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493210675670016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8846717891942 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553222952026112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12307378580480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562806433179136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553222926598144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493210758805504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89274578 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8849821361074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8864802533376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44390718 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74249423 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚571668130305536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32551022 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884321877101576e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12307433581568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562806459570688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74249417 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚571671262104064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8861697126400 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163547133 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89297710 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚562072773387186 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚562069644490086 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89297710 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚4149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44390715 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163547133 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562806459570688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76942245 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76941737 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4166.065924605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89274578 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚578217990204928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32925747 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚4199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486411457566208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚568516570899670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚578214858419712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚578217990204928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75096339 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚486411429431808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8965214919168 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486411457566208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚568516570899670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚578214858419712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚578217990204928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75096339 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚486411429431808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8965214919168 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚578217990204928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968Ö‰Íz,2ª- _wandbruntime‚4212  _runtime‚4212 °² '«ƒN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90332430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486523840295936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚568648076694194 trainer/global_step‚4199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75113617 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569381137588736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90355562 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8952171138150 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75113617 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8955274607282 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569381163980288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77836549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4212.494017321 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165469185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559692802086912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8967327233024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44903368 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8967327233024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚568644947797094 $ _timestamp‚1.788432234138614e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8970432640000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8970432640000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165469185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165446047 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 _step‚343 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77836014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚486523868430336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569381137588736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32932649 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578351596620288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578351596620288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569381163980288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32933181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75113623 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90332430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486523840295936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚568648076694194 trainer/global_step‚4199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75113617 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569381137588736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90355562 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8952171138150 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75113617 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8955274607282 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569381163980288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77836549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4212.494017321 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165469185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559692802086912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8967327233024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44903368 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8967327233024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚568644947797094 $ _timestamp‚1.788432234138614e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8970432640000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.sm6°j9»all_pool.allocated‚ 8970432640000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165469185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165446047 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 _step‚343 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77836014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚486523868430336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569381137588736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32932649 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578351596620288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578351596620288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569381163980288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32933181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75113623 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584895192935936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91392298 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575091874206678 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575088745309786 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9073950419456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75960533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75960533 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚575824374301696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75960539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33304957 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33304428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492032000772608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566033255146496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚575824374301696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9073950419456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75960533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75960533 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚575824374301696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75960539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33304957 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33304428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492032000772608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566033255146496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚575824374301696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9073950419456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584895192935936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91392298 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575091874206678 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575088745309786 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9073950419456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Úƒw¢à2¼- _wandbruntime‚4258  _runtime‚4258T„ås½„N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9072957339648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504745681060352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75977823 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492144383502336 _runtime‚4258.712208233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45416018 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9076062746624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9072957339648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167368099 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78727923 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.alú?—l.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167391237 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91390282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566162652147712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9076062746624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75977823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9057624384358 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585028799337984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167391237 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585031931136512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75977817 % _timestamp‚1.7884322803568277e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33311908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45416015 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12601214422528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9060727853490 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91413414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33312434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12601269423616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167368099 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91413414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585031931136512 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚575955868389888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚575955868389888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78728452 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75977817 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚575955841998336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575220251104102 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45416015 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12601214422528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9060727853490 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91413414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33312434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12601269423616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167368099 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91413414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585031931136512 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚575955868389888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚575955868389888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78728452 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75977817 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚575955841998336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575220251104102 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75977823 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492144383502336 _runtime‚4258.712208233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45416018 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9076062746624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9072957339648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167368099 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78727923 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167391237 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91390282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566162652147712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984Û÷;ª2¿- _wandbruntime‚4258  _runtime‚4258bM±§VÀÐ 1loss/FBt1/assignment_loss‚0.05566329408601678 1loss/FBt4/assignment_loss‚0.19250785785382504 /loss/FRt4/assignment_loss‚4.447505295276642 %loss/total_loss‚46.56238219045067 epoch‚3 .loss/FRt2/assignment_loss‚4.23481624357162 1loss/SRqqt4/assignment_loss‚2.391755578979369 trainer/global_step‚4249 .loss/FRt2/detection_loss‚2.527451907434771 0loss/FRt3/assignment_loss‚2.0909732197815516 /loss/FBt2/assignment_loss‚0.586078009475015 /loss/FRt4/detection_loss‚2.7779186385519363 /loss/FBt1/detection_loss‚0.7466785836149938 .loss/FBt3/detection_loss‚3.083665524919828 /loss/FBt4/detection_loss‚1.6493092253804207 _runtime‚ 4258.71329336 1loss/SRqqt2/detection_loss‚3.1489927846452463 1loss/SRqqt3/detection_loss‚1.3003428500405578 1loss/SRqqt4/detection_loss‚2.1659005399673217 /loss/FBt2/detection_loss‚2.3357433319091796 2loss/SRqqt3/assignment_loss‚1.4125782764451795 1loss/FBt3/assignment_loss‚0.20762935487437062 /loss/FRt1/detection_loss‚2.4885451414045834 .loss/FRt3/detection_loss‚2.156088093401126 1loss/SRqqt1/detection_loss‚1.4108787973300885 % _timestamp‚1.7884322803583426e+09 _step‚348 .loss/FRt1/assignment_loss‚2.35700752053942 2loss/SRqqt1/assignment_loss‚1.3380451836954739 1loss/SRqqt2/assignment_loss‚1.456306937272134ËÊ»2[ÁÕ .loss/FRt2/assignment_loss‚4.23481624357162 1loss/SRqqt4/assignment_loss‚2.391755578979369 trainer/global_step‚4249 .loss/FRt2/detection_loss‚2.527451907434771 0loss/FRt3/assignment_loss‚2.0909732197815516 /loss/FBt2/assignment_loss‚0.586078009475015 /loss/FRt4/detection_loss‚2.7779186385519363 /loss/FBt1/detection_loss‚0.7466785836149938 .loss/FBt3/detection_loss‚3.083665524919828 /loss/FBt4/detection_loss‚1.6493092253804207 _runtime‚ 4258.71329336 1loss/SRqqt2/detection_loss‚3.1489927846452463 1loss/SRqqt3/detection_loss‚1.3003428500405578 1loss/SRqqt4/detection_loss‚2.1659005399673217 /loss/FBt2/detection_loss‚2.3357433319091796 2loss/SRqqt3/assignment_loss‚1.4125782764451795 1loss/FBt3/assignment_loss‚0.20762935487437062 /loss/FRt1/detection_loss‚2.4885451414045834 .loss/FRt3/detection_loss‚2.156088093401126 1loss/SRqqt1/detection_loss‚1.4108787973300885 % _timestamp‚1.7884322803583426e+09 _step‚348 .loss/FRt1/assignment_loss‚2.35700752053942 2loss/SRqqt1/assignment_loss‚1.3380451836954739 1loss/SRqqt2/assignment_loss‚1.456306937272134 1loss/FBt1/assignment_loss‚0.05566329408601678 1loss/FBt4/assignment_loss‚0.19250785785382504 /loss/FRt4/assignment_loss‚4.447505295276642 %loss/total_loss‚46.56238219045067 epoch‚3Ü4 ì}žÂ‚(:“ ™ŸåÔø͵ gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.462199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G;wÂ(:ì ™ŸåÔ­¿…¸ proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 2.146484375# disk.dm-0.out‚34686.78955078125$ memory_percent‚94.03266906738281$ proc.memory.availableMB‚977.6875 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.sda.in‚0.203125* proc.memory.percent‚18.000316619873047 disk.nvme0n1.in‚ 387.80859375 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.dm-0.in‚ 387.80859375 disk.drbd2080.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚34686.78955078125 cpu‚25.252497951214142 network.recv‚ 1.541322e+06% disk./.usageGB‚278.41928482055664 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 158.76171875 disk.sda1.out‚0! proc.memory.rssMB‚ 2949.171875* disk./.usagePercent‚18.684951813647487 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚34686.78955078125 disk.dm-1.out‚ 158.76171875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2080.out‚0 network.sent‚ 9.953036e+06 disk.sda.out‚ 158.76171875sº!“Ä‚(:ˆ ¨ŸåÔ›–ø´ gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚161.373' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø"¿LzÅ‚(:ï ¨ŸåÔêÖ‰¸ disk.rbd0.out‚0$ memory_percent‚94.03576850891113' proc.memory.availableMB‚ 977.1796875 disk.sda2.out‚0 disk.dm-1.out‚ 159.37109375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 cpu‚25.251733418014663 proc.cpu.threads‚17* disk./.usagePercent‚18.684990982800226 disk.drbd2080.in‚0* proc.memory.percent‚18.000316619873047 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 2.146484375 disk.dm-0.in‚ 387.80859375 disk.sda1.in‚0 network.sent‚ 1.0061012e+07 network.recv‚ 1.547266e+06 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.sda.out‚ 159.37109375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2949.171875 disk.nvme0n1.in‚ 387.80859375( disk.nvme0c0n1.out‚34689.37353515625# disk.dm-0.out‚34689.37353515625 disk.sda3.out‚ 159.37109375 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0$ disk./.usageGB‚278.4198684692383& disk.nvme0n1.out‚34689.37353515625÷¼Ž¨žÆ‚(:“ ·ŸåÔ’»¥µ gpu.0.gpu‚69.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.004200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00êì¼sÇ‚(:è ·ŸåÔí¿¦· disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 network.sent‚ 1.0066124e+07" disk.nvme0c0n1.in‚ 387.80859375( disk.nvme0c0n1.out‚34690.60205078125# disk.dm-0.out‚34690.60205078125 disk.sda1.out‚0) disk./.usagePercent‚18.68503655214132% disk./.usageGB‚278.42054748535156 disk.sda2.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 160.015625! proc.memory.rssMB‚ 2949.171875 disk.dm-0.in‚ 387.80859375 disk.sda3.out‚ 160.015625 disk.drbd1716.in‚0 disk.drbd2080.in‚0& proc.memory.availableMB‚ 977.609375* proc.memory.percent‚18.000316619873047 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0$ memory_percent‚94.03314590454102 disk.rbd0.out‚0 disk.nvme0n1.in‚ 387.80859375 disk.dm-1.out‚ 160.015625 cpu‚25.251099435548625 network.recv‚ 1.551221e+06 disk.rbd0.in‚0& disk.nvme0n1.out‚34690.60205078125 disk.sda2.out‚0 disk.sda3.in‚0.203125 proc.cpu.threads‚17 disk.rbd1.out‚ 2.232421875 disk.sda1.in‚0 disk.drbd2265.out‚0ªÔ²ñ2È- _wandbruntime‚4304  _runtime‚4304Ò‘ÉÉà ' lr-AdamW/pg1‚0.00016649344784642114 ' lr-AdamW/pg2‚0.00016649344784642114 trainer/global_step‚4299 % _timestamp‚1.7884323259543805e+09 _runtime‚4304.309532842 _step‚349šâËFÎÊÈ % _timestamp‚1.7884323259543805e+09 _runtime‚4304.309532842 _step‚349 ' lr-AdamW/pg1‚0.00016649344784642114 ' lr-AdamW/pg2‚0.00016649344784642114 trainer/global_step‚4299ÝÞ£+»2Ë- _wandbruntime‚4304  _runtime‚4304ìFÙÌúO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9179580526080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497652515844608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12745269097984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169274889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚572503105207296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚4304.310380811 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚350 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79604500 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169251777 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76824739 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79605022 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚572503079779328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92427044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚510397784942592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582399078711296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582399052319744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9179580526080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚497652543979008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚581664048616794 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9160968837466 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9176475132416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inaë†yø' ctive_split.small_pool.allocated‚33686607 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76824733 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92427044 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45918415 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚510397868091392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591575527452160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591575527452160 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚350 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79604500 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169251777 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76824739 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79605022 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚572503079779328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92427044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚510397784942592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582399078711296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582399052319744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9179580526080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚497652543979008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚581664048616794 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9160968837466 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9176475132416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33686607 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76824733 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92427044 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45918415 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚510397868091392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591575527452160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591575527452160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _runtime‚4305.194885125 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169290151 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582530546407936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45928665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚582530546407936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76842017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313289 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚572632502208512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92448134 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92471266 % _timestamp‚1.7884323268395162e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169290151 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497764954843136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚591712265652736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92471266 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚572632476780544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚510513155261440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚581798683308210 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9181692853248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581795554411110 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚4299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚497764926708736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12748228552704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76842017 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9178587446272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _step‚351 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9178587446272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚591709133854208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582530572799488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9181692853248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79622945 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9163077630566 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313289 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45928668 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9166181099698 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76842023 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚591712265652736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚500 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79622445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚510513236299776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0hÁúF`Ò‰N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33693780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92471266 % _timestamp‚1.7884323268395162e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169290151 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497764954843136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚591712265652736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92471266 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚572632476780544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚510513155261440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚581798683308210 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9181692853248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581795554411110 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚4299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚497764926708736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12748228552704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76842017 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9178587446272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _step‚351 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9178587446272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚591709133854208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582530572799488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9181692853248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79622945 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9163077630566 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313289 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45928668 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9166181099698 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76842023 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚591712265652736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚500 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79622445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚510513236299776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _runtime‚4305.194885125 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169290151 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582530546407936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45928665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚582530546407936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76842017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Bæ¯ d/oversize_segments.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313289 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚572632502208512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92448134 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12892167262720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚354 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77688939 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80498948 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588973783120896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77688939 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588973783120896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93508002 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34067886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚588973756729344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34068435 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚503273087185408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚598258993753600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93484896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9285210632704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598255861968384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46431062 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9285210632704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93508002 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516165171299328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚503273059051008 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚578972955268096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12892167262720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚354 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77688939 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80498948 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588973783120896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77688939 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588973783120896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93508002 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34067886 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚588973756729344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34068435 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚503273087185408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚598258993753600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93484896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9285210632704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598255861968384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46431062 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9285210632704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93508002 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516165171299328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚503273059051008 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚578972955268096 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚588370857718118 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 trainer/global_step‚4349 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589105277209088 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚598392600168960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9268530876774 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9287322959872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171235341 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93505986 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516280597066752 RCDeviceStatsMonitor.on_t/È{ rain_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚579102352269312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46441318 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93529118 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34075956 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171212203 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171235341 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171212203 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77706223 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589105250817536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚588373986615218 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589105277209088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77706223 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80516740 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93529118 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9271634345906 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4351.676760253 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93505986 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976—=r'ä‰N EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171212203 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171235341 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171212203 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77706223 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589105250817536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚588373986615218 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589105277209088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77706223 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80516740 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93529118 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9271634345906 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4351.676760253 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93505986 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚588370857718118 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 trainer/global_step‚4349 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589105277209088 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚598392600168960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9268530876774 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9287322959872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171235341 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93505986 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516280597066752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚579102352269312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46441318 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93529118 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34075956 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180ãcÉDÎ2å- _wandbruntime‚4351  _runtime‚4351äîëXæÒ 2loss/SRqqt3/assignment_loss‚1.2146806350598733 2loss/FBt1/assignment_loss‚0.060127247051766336 0loss/FBt3/assignment_loss‚0.1487046065949471 /loss/FRt1/detection_loss‚1.7502239381428808 .loss/FRt2/detection_loss‚2.773953399487904 .loss/FRt3/detection_loss‚2.322266097773205 /loss/FBt1/detection_loss‚0.9669695360782115 &loss/total_loss‚46.348558388764246 2loss/SRqqt1/assignment_loss‚1.1968057134188712 0loss/SRqqt4/assignment_loss‚2.02264158187374 1loss/SRqqt3/detection_loss‚1.6410280275675986 /loss/FBt4/detection_loss‚1.5995180604082566 .loss/FRt4/detection_loss‚3.671154390363132 /loss/FRt2/assignment_loss‚4.490874290466309 1loss/FBt2/assignment_loss‚0.16112622886395492 _runtime‚ 4351.6779197 epoch‚3 trainer/global_step‚4349 /loss/FRt4/assignment_loss‚3.778164162355311 0loss/FBt4/assignment_loss‚0.2270724209577695 1loss/SRqqt1/detection_loss‚1.0984762396663428 1loss/SRqqt2/detection_loss‚2.9007919482562854 /loss/FRt3/assignment_loss‚2.363524916497144 % _timestamp‚1.7884323733229678e+09 _step‚356 /loss/FRt1/assignment_loss‚2.072027148678899 2loss/SRqqt2/assignment_loss‚2.3941828193871872 1loss/SRqqt4/detection_loss‚2.4298826494524555 /loss/FBt2/detection_loss‚2.7469624876976013 /loss/FBt3/detection_loss‚2.3173998426645994Õ€¤]ç× % _timestamp‚1.7884323733229678e+09 _step‚356 /loss/FRt1/assignment_loss‚2.072027148678899 2loss/SRqqt2/assignment_loss‚2.3941828193871872 1loss/SRqqt4/detection_loss‚2.4298826494524555 /loss/FBt2/detection_loss‚2.7469624876976013 /loss/FBt3/detection_loss‚2.3173998426645994 2loss/SRqqt3/assignment_loss‚1.2146806350598733 2loss/FBt1/assignment_loss‚0.060127247051766336 0loss/FBt3/assignment_loss‚0.1487046065949471 /loss/FRt1/detection_loss‚1.7502239381428808 .loss/FRt2/detection_loss‚2.773953399487904 .loss/FRt3/detection_loss‚2.322266097773205 /loss/FBt1/detection_loss‚0.9669695360782115 &loss/total_loss‚46.348558388764246 2loss/SRqqt1/assignment_loss‚1.1968057134188712 0loss/SRqqt4/assignment_loss‚2.02264158187374 1loss/SRqqt3/detection_loss‚1.6410280275675986 /loss/FBt4/detection_loss‚1.5995180604082566 .loss/FRt4/detection_loss‚3.671154390363132 /loss/FRt2/assignment_loss‚4.490874290466309 1loss/FBt2/assignment_loss‚0.16112622886395492 _runtime‚ 4351.6779197 epoch‚3 trainer/global_step‚4349 /loss/FRt4/assignment_loss‚3.778164162355311 0loss/FBt4/assignment_loss‚0.2270724209577695 1loss/SRqqt1/detection_loss‚1.0984762396663428 1loss/SRqqt2/detection_loss‚2.9007919482562854 /loss/FRt3/assignment_loss‚2.363524916497144äì “è‚(:’ ‚ åÔÍ㤵 gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚303.642' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.72840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B]´]yé‚(:î ‚ åÔ£€Ã·* disk./.usagePercent‚18.685371409996108 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 2.39013671875 disk.dm-0.in‚ 387.80859375 disk.sda3.in‚0.203125 disk.sda2.in‚0% disk.nvme0n1.out‚34699.3916015625 disk.dm-1.out‚ 162.91796875 disk.rbd1.in‚0$ memory_percent‚94.03829574584961 proc.cpu.threads‚17 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2949.65234375 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 387.80859375 disk.sda1.in‚0 cpu‚25.247547798700616 network.sent‚ 1.0278744e+07 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚34699.3916015625 disk.sda3.out‚ 162.91796875 disk.drbd1716.in‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0& proc.memory.availableMB‚ 976.765625* proc.memory.percent‚18.003249168395996 network.recv‚ 1.577144e+06# disk./.usageGB‚278.425537109375 disk.sda.out‚ 162.91796875 disk.nvme0n1.in‚ 387.80859375 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0" disk.dm-0.out‚34699.3916015625ÀXá=ê‚(:’ ‘ åÔÌ•µ gpu.0.gpu‚53.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚302.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.46699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›";Þwë‚(:ì ‘ åÔ„Žµ·$ disk./.usageGB‚278.4261360168457 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2949.65234375 disk.sda3.out‚ 163.51171875 disk.sda1.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 163.51171875 disk.rbd1.out‚ 2.39013671875" disk.dm-0.out‚34700.3603515625 cpu‚25.2468197967366 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 387.80859375% disk.nvme0n1.out‚34700.3603515625 disk.drbd2266.in‚0 network.recv‚ 1.584818e+06" disk.nvme0c0n1.in‚ 387.80859375& proc.memory.availableMB‚ 976.859375 disk.sda.out‚ 163.51171875 network.sent‚ 1.038784e+07 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚34700.3603515625 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0$ memory_percent‚94.03772354125977* proc.memory.percent‚18.003249168395996 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 387.80859375 proc.cpu.threads‚17* disk./.usagePercent‚18.685411603178984\„“ì‚(:ˆ   åÔ¨ŒÌµ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.9654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ït¡ yí‚(:î   åÔãò—· cpu‚25.24615475295953 disk.sda.out‚ 163.94921875 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 163.94921875 disk.sda2.in‚0* proc.memory.percent‚18.003249168395996& disk.nvme0n1.out‚34702.13720703125 disk.rbd1.out‚2.453125 disk.dm-0.in‚ 387.80859375 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 387.80859375 disk.drbd2266.out‚0 disk.sda3.out‚ 163.94921875 disk.sda1.out‚0$ memory_percent‚94.03462409973145" disk.nvme0c0n1.in‚ 387.80859375( disk.nvme0c0n1.out‚34702.13720703125# disk.dm-0.out‚34702.13720703125 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 proc.cpu.threads‚17% disk./.usageGB‚278.42684173583984 disk.rbd1.in‚0* disk./.usagePercent‚18.685458964572817 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 977.3671875# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 1.0393171e+07 network.recv‚ 1.588306e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda2.out‚0(©ÄÓ2î- _wandbruntime‚4397  _runtime‚4397w¤ÇÉïà ' lr-AdamW/pg1‚0.00016416815553555283 ' lr-AdamW/pg2‚0.00016416815553555283 trainer/global_step‚4399 % _timestamp‚1.7884324186923938e+09 _runtime‚4397.047538804 _step‚357eP™:ÎðÈ ' lr-AdamW/pg1‚0.00016416815553555283 ' lr-AdamW/pg2‚0.00016416815553555283 trainer/global_step‚4399 % _timestamp‚1.7884324186923938e+09 _runtime‚4397.047538804 _step‚357å°Ùöð2ñ- _wandbruntime‚4397  _runtime‚4397Y$IîòúO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81394794 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13038811448320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521932496854528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13038866462720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9387735345664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604936196484608 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚585442779900928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚595548487530496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94542748 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81395326 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34451611 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94542748 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9390840739328 % _timestamp‚1.7884324186930907e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9371875329882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46943715 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9390840739328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508893630391808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173118993 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚604936196484608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_trainU-ÊÕ_batch_start/segment.large_pool.freed‚0 _step‚358 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9387735345664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚595548487530496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34451082 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604939328269824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521932413705728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78553139 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78553133 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78553139 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78553133 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94565854 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302‡Ã¹(óÿO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173095881 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚358 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9387735345664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚595548487530496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34451082 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604939328269824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521932413705728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78553139 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78553133 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78553139 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78553133 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94565854 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81394794 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13038811448320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521932496854528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13038866462720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9387735345664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604936196484608 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚585442779900928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚595548487530496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94542748 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81395326 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34451611 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94542748 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9390840739328 % _timestamp‚1.7884324186930907e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9371875329882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46943715 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9390840739328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508893630391808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173118993 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚604936196484608 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173134255 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚594946161025126 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9373984122982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78570417 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595679955227136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚605072934685184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚605069802886656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173157393 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509006041255936 _runtime‚4397.932115665 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94586970 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81412553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚522047766109696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78570423 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46953965 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13041807989248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94563838 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚595679955227136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚595679981618688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78570423 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚605072934685184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚522047849245184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚509006013121536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81413066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34459098 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594949289922226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78570417 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94586970Œfæ‹!öˆN WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚605072934685184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚522047849245184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚509006013121536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81413066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34459098 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594949289922226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78570417 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94586970 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9392953066496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173134255 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94563838 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173134255 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚594946161025126 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9373984122982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78570417 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595679955227136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚605072934685184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚605069802886656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173157393 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509006041255936 _runtime‚4397.932115665 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94586970 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81412553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚522047766109696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78570423 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46953965 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13041807989248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GA «jƒDeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94563838 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚595679955227136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚595679981618688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78570423 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0çX7Äe2÷- _wandbruntime‚4397  _runtime‚4397D+2ûXøÒ /loss/FBt4/detection_loss‚1.7819084094630346 .loss/FRt2/assignment_loss‚3.69532372095646 0loss/FRt4/assignment_loss‚4.2832370458110685 2loss/SRqqt1/assignment_loss‚1.3107000966664664 1loss/SRqqt2/assignment_loss‚3.378018217928269 0loss/FBt3/assignment_loss‚0.2075454536670198 1loss/SRqqt3/detection_loss‚1.6348660751956248 /loss/SRqqt4/detection_loss‚2.49281773184027 2loss/SRqqt3/assignment_loss‚1.4743811294310174 0loss/FBt1/assignment_loss‚0.0650831052485462 0loss/FBt2/assignment_loss‚0.2629061227567391 0loss/FRt1/assignment_loss‚2.0201195031404495 /loss/FRt1/detection_loss‚1.9300866513513029 trainer/global_step‚4399 % _timestamp‚1.7884324195782483e+09 /loss/FBt1/detection_loss‚0.7933147162845244 1loss/FBt4/assignment_loss‚0.17576160384500186 /loss/FRt2/detection_loss‚2.6883735205882635 /loss/FRt3/detection_loss‚2.3193930167218912 /loss/FRt4/detection_loss‚2.7926553949233024 .loss/FBt3/detection_loss‚2.398538961013158 0loss/FRt3/assignment_loss‚1.9743743862794794 /loss/FBt2/detection_loss‚2.5575625517151575 2loss/SRqqt4/assignment_loss‚1.8934834022074938 0loss/SRqqt1/detection_loss‚1.071962070153203 1loss/SRqqt2/detection_loss‚3.1050072347416595 epoch‚3 _runtime‚4397.933186194 _step‚360 $loss/total_loss‚46.3074201219294$.µ€]ù× trainer/global_step‚4399 % _timestamp‚1.7884324195782483e+09 /loss/FBt1/detection_loss‚0.7933147162845244 1loss/FBt4/assignment_loss‚0.17576160384500186 /loss/FRt2/detection_loss‚2.6883735205882635 /loss/FRt3/detection_loss‚2.3193930167218912 /loss/FRt4/detection_loss‚2.7926553949233024 .loss/FBt3/detection_loss‚2.398538961013158 0loss/FRt3/assignment_loss‚1.9743743862794794 /loss/FBt2/detection_loss‚2.5575625517151575 2loss/SRqqt4/assignment_loss‚1.8934834022074938 0loss/SRqqt1/detection_loss‚1.071962070153203 1loss/SRqqt2/detection_loss‚3.1050072347416595 epoch‚3 _runtime‚4397.933186194 _step‚360 $loss/total_loss‚46.3074201219294 /loss/FBt4/detection_loss‚1.7819084094630346 .loss/FRt2/assignment_loss‚3.69532372095646 0loss/FRt4/assignment_loss‚4.2832370458110685 2loss/SRqqt1/assignment_loss‚1.3107000966664664 1loss/SRqqt2/assignment_loss‚3.378018217928269 0loss/FBt3/assignment_loss‚0.2075454536670198 1loss/SRqqt3/detection_loss‚1.6348660751956248 /loss/SRqqt4/detection_loss‚2.49281773184027 2loss/SRqqt3/assignment_loss‚1.4743811294310174 0loss/FBt1/assignment_loss‚0.0650831052485462 0loss/FBt2/assignment_loss‚0.2629061227567391 0loss/FRt1/assignment_loss‚2.0201195031404495 /loss/FRt1/detection_loss‚1.9300866513513029è1r·“ú‚(:ˆ ¯ åÔ„ý×µ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.4252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¤BEwû‚(:ì ¯ åÔ´âí· disk.nvme0n1.in‚ 387.80859375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 164.66015625 disk.drbd2265.in‚0 disk.sda3.in‚0.203125 disk.rbd0.in‚0 network.recv‚ 1.592574e+06* disk./.usagePercent‚18.685494293612543 disk.dm-1.in‚0.203125 disk.sda2.out‚0$ memory_percent‚94.03619766235352 disk.dm-0.in‚ 387.80859375 disk.sda3.out‚ 164.66015625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚34703.03173828125 disk.sda2.in‚0* proc.memory.percent‚18.003249168395996 cpu‚25.24542702666179 disk.rbd1.out‚2.453125 network.sent‚ 1.0399137e+07 disk.dm-1.out‚ 164.66015625 disk.drbd2266.in‚0 disk.drbd2266.out‚0& proc.memory.availableMB‚ 977.109375# proc.memory.rssMB‚ 2949.65234375( disk.nvme0c0n1.out‚34703.03173828125# disk.dm-0.out‚34703.03173828125 disk.drbd1716.in‚0 disk.sda.in‚0.203125$ disk./.usageGB‚278.4273681640625nÝâ“ü‚(:ˆ ¾ åÔ𿊵 gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚211.376' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.2752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02yáwý‚(:ì ¾ åÔ¬êϸ disk.drbd2266.out‚0% disk./.usageGB‚278.42821502685547 disk.sda3.in‚0.203125 disk.sda3.out‚ 165.01953125 disk.drbd2265.out‚0 cpu‚25.24475966721779 disk.dm-0.in‚ 387.80859375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚34704.95654296875 disk.dm-1.in‚0.203125 disk.sda2.in‚0% proc.memory.availableMB‚ 976.90625 network.sent‚ 1.0507671e+07# disk.dm-0.out‚34704.95654296875 disk.sda1.in‚0 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 387.80859375# proc.memory.rssMB‚ 2949.65234375 disk.drbd2266.in‚0 network.recv‚ 1.599432e+06 disk.drbd1716.in‚0 disk.sda.in‚0.203125( disk.nvme0c0n1.out‚34704.95654296875* disk./.usagePercent‚18.685551127285144 disk.drbd2080.in‚0 disk.sda.out‚ 165.01953125 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 387.80859375 disk.rbd1.out‚2.453125$ memory_percent‚94.03743743896484* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.dm-1.out‚ 165.01953125Bî“þ‚(:ˆ Í åÔÔ®¯µ gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚158.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05uëÃrÿ‚(:ç Í åÔè„£· disk.drbd2265.out‚0* proc.memory.percent‚18.003249168395996* disk./.usagePercent‚18.685803038698836 disk.sda3.out‚ 165.640625 disk.sda.in‚0.203125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2949.65234375# disk.dm-0.out‚34706.56689453125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 cpu‚25.24412110672565 disk.drbd2266.in‚0 disk.dm-0.in‚ 387.80859375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 165.640625 disk.drbd2265.in‚0& proc.memory.availableMB‚ 976.703125 network.recv‚ 1.604404e+06% disk./.usageGB‚278.43196868896484 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 387.80859375& disk.nvme0n1.out‚34706.56689453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 165.640625 disk.sda3.in‚0.203125 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd1.out‚2.453125( disk.nvme0c0n1.out‚34706.56689453125 disk.nvme0n1.in‚ 387.80859375$ memory_percent‚94.03867721557617 network.sent‚ 1.0513969e+07 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0ô« 2€- _wandbruntime‚4443  _runtime‚4443š»ýLÈ ' lr-AdamW/pg2‚0.00016298604308005894 trainer/global_step‚4449 $ _timestamp‚1.788432465172928e+09 _runtime‚4443.527982247 _step‚361 ' lr-AdamW/pg1‚0.000162986043080058940öÃRÍ‚Ç _runtime‚4443.527982247 _step‚361 ' lr-AdamW/pg1‚0.00016298604308005894 ' lr-AdamW/pg2‚0.00016298604308005894 trainer/global_step‚4449 $ _timestamp‚1.788432465172928e+09éaƒ2A2ƒ- _wandbruntime‚4443  _runtime‚4443hGîÿ'„ùO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9493365452288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95600600 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34831901 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527699707692544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591912629961728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79417333 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175017933 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚611619662786048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175041045 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514514145463808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚602123165548544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47456365 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13185562228736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9480432045014 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47456362 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚611616531000832 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚601393087434710 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591912655389696 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79417339 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚602123191940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9493365452288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚602123191940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9496470845952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95600600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79417339 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82287715 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 % _timestamp‚1.7884324651736972e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9496470845952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611616531000832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34831353 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302AE£7(…þO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47456362 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚611616531000832 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚601393087434710 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591912655389696 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79417339 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚602123191940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9493365452288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚602123191940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9496470845952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95600600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79417339 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82287715 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 % _timestamp‚1.7884324651736972e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9496470845952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611616531000832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34831353 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9493365452288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95600600 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34831901 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527699707692544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591912629961728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79417333 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175017933 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚611619662786048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175041045 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514514145463808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚602123165548544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47456365 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13185562228736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9480432045014 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180êÍ)¹á2†- _wandbruntime‚4444  _runtime‚4444Ž ÓÑ ‡ƒN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95621690 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175056307 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 $ _timestamp‚1.788432466057061e+09 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_po¬þ#8ol.freed‚ 9495477766144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82306165 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚592042026962944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79434623 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611750137402880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95621690 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚527815038999040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592042052390912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95644822 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95644822 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527815124231680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4449 _step‚363 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47466618 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚602254659636736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175056307 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13188539769344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9479437369190 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82305629 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175079445 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175079445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79434623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9498583173120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9495477766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79434617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13188482671104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚601521464332134 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34839547 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9498583173120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚611753269201408ªa'ˆˆN NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79434617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13188482671104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚601521464332134 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34839547 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9498583173120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚611753269201408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95621690 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175056307 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 $ _timestamp‚1.788432466057061e+09 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9495477766144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82306165 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚592042026962944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79434623 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611750137402880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95621690 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚527815038999040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592042052390912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95644822 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95644822 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527815124231680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4449 _step‚363 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47466618 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚602254659636736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175056307 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13188539769344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9479437369190 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82305629 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175079445 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175079445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79434623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9498583173120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9495477766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6ë(…4Z2‰- _wandbruntime‚4444  _runtime‚4444ëÄõÊYŠÓ /loss/FRt2/detection_loss‚2.6456920730101094 /loss/FBt2/detection_loss‚2.1829075548383923 2loss/SRqqt4/assignment_loss‚1.9840967687419975 /loss/FBt3/assignment_loss‚0.517925399186259 /loss/FBt3/detection_loss‚2.3681446530602197 % _timestamp‚1.7884324660586774e+09 /loss/FBt4/detection_loss‚1.4081715266737673 epoch‚3 /loss/FRt1/assignment_loss‚2.422641470151789 0loss/FRt4/assignment_loss‚4.2439715678875265 /loss/SRqqt1/detection_loss‚1.19553749150399 1loss/SRqqt3/detection_loss‚0.9230777486524684 _runtime‚4444.413626797 _step‚364 2loss/SRqqt1/assignment_loss‚1.3243620252857606 2loss/SRqqt3/assignment_loss‚1.2479592628697767 1loss/FBt1/assignment_loss‚0.11461231370224369 0loss/FBt4/assignment_loss‚0.2518629043618683 trainer/global_step‚4449 0loss/FBt2/assignment_loss‚0.6298496293812705 0loss/SRqqt4/detection_loss‚2.646833440432182 .loss/FRt3/detection_loss‚2.320622358309186 1loss/SRqqt2/detection_loss‚2.6482620988573347 /loss/FBt1/detection_loss‚0.9929851257801056 &loss/total_loss‚46.006253742245406 0loss/FRt3/assignment_loss‚2.0113511111425315 1loss/SRqqt2/assignment_loss‚2.145246116604124 /loss/FRt1/detection_loss‚1.9180055610178148 .loss/FRt4/detection_loss‚4.016718804836273 0loss/FRt2/assignment_loss‚3.8454167359584086¼¤çÌ^‹Ø /loss/FRt1/detection_loss‚1.9180055610178148 .loss/FRt4/detection_loss‚4.016718804836273 0loss/FRt2/assignment_loss‚3.8454167359584086 /loss/FRt2/detection_loss‚2.6456920730101094 /loss/FBt2/detection_loss‚2.1829075548383923 2loss/SRqqt4/assignment_loss‚1.9840967687419975 /loss/FBt3/assignment_loss‚0.517925399186259 /loss/FBt3/detection_loss‚2.3681446530602197 % _timestamp‚1.7884324660586774e+09 /loss/FBt4/detection_loss‚1.4081715266737673 epoch‚3 /loss/FRt1/assignment_loss‚2.422641470151789 0loss/FRt4/assignment_loss‚4.2439715678875265 /loss/SRqqt1/detection_loss‚1.19553749150399 1loss/SRqqt3/detection_loss‚0.9230777486524684 _runtime‚4444.413626797 _step‚364 2loss/SRqqt1/assignment_loss‚1.3243620252857606 2loss/SRqqt3/assignment_loss‚1.2479592628697767 1loss/FBt1/assignment_loss‚0.11461231370224369 0loss/FBt4/assignment_loss‚0.2518629043618683 trainer/global_step‚4449 0loss/FBt2/assignment_loss‚0.6298496293812705 0loss/SRqqt4/detection_loss‚2.646833440432182 .loss/FRt3/detection_loss‚2.320622358309186 1loss/SRqqt2/detection_loss‚2.6482620988573347 /loss/FBt1/detection_loss‚0.9929851257801056 &loss/total_loss‚46.006253742245406 0loss/FRt3/assignment_loss‚2.0113511111425315 1loss/SRqqt2/assignment_loss‚2.145246116604124ìƒJ„‘Œ‚(:† Ü åÔÂŽŒµ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚293.47' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶È«z‚(:ï Ü åÔù±™· network.recv‚ 1.608288e+06( disk.nvme0c0n1.out‚34712.96142578125 disk.sda3.out‚ 166.09765625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 166.09765625 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0' proc.memory.availableMB‚ 976.6640625# proc.memory.rssMB‚ 2949.65234375 disk.sda2.out‚0 network.sent‚ 1.0519389e+07 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125% disk./.usageGB‚278.43239974975586" disk.nvme0c0n1.in‚ 387.80859375 disk.dm-0.in‚ 387.80859375 disk.sda.out‚ 166.09765625 disk.rbd0.in‚0& disk.nvme0n1.out‚34712.96142578125* disk./.usagePercent‚18.685831967550207 disk.nvme0n1.in‚ 387.80859375 disk.rbd1.out‚2.453125* proc.memory.percent‚18.003249168395996 disk.sda3.in‚0.203125 disk.rbd0.out‚0# disk.dm-0.out‚34712.96142578125 disk.sda1.in‚0 disk.drbd2080.out‚0$ memory_percent‚94.03891563415527 cpu‚25.243472115185178 proc.cpu.threads‚17Ñ|-’Ž‚(:‡ ë åÔŽñ¹µ gpu.0.gpu‚78.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚271.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.257 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä9zwq‚(:æ ë åÔ½žÞ· disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 166.734375 disk.sda1.out‚0 disk.drbd2080.out‚0$ disk./.usageGB‚278.4341583251953 disk.dm-1.out‚ 166.734375 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚34713.64892578125# proc.memory.rssMB‚ 2949.65234375 disk.rbd1.out‚2.453125 disk.drbd1716.in‚0& disk.nvme0n1.out‚34713.64892578125( proc.memory.availableMB‚ 976.91796875* proc.memory.percent‚18.003249168395996 disk.sda.in‚0.203125 disk.rbd0.in‚0 network.recv‚ 1.615511e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 387.80859375 disk.rbd0.out‚0 disk.sda2.in‚0) disk./.usagePercent‚18.68594998702349 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 166.734375 disk.nvme0n1.in‚ 387.80859375$ memory_percent‚94.03736591339111 cpu‚25.24276476557694" disk.nvme0c0n1.in‚ 387.80859375# disk.dm-0.out‚34713.64892578125 disk.drbd2080.in‚0 disk.sda2.out‚0 network.sent‚ 1.062804e+07 disk.dm-1.in‚0.203125õ“F‚(:’ ú åÔíí³µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.583' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.31660000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p•Õt‘‚(:é ú åÔ¯íÒ¶ disk.sda2.in‚0 disk.dm-1.out‚ 167.2265625" disk.dm-0.out‚34715.1435546875# proc.memory.rssMB‚ 2949.65234375 network.recv‚ 1.619735e+06 disk.drbd2266.out‚0 disk.rbd1.out‚2.453125' disk.nvme0c0n1.out‚34715.1435546875 disk.sda1.out‚0 network.sent‚ 1.0633932e+07 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.dm-0.in‚ 387.80859375* disk./.usagePercent‚18.685987620131023% disk.nvme0n1.out‚34715.1435546875 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.003249168395996 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 167.2265625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 167.2265625 disk.drbd2265.in‚0$ memory_percent‚94.03736591339111% disk./.usageGB‚278.43471908569336 disk.rbd0.out‚0 disk.nvme0n1.in‚ 387.80859375 disk.sda2.out‚0( proc.memory.availableMB‚ 976.91796875 cpu‚25.24213642664059 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 387.80859375 disk.sda3.in‚0.203125 disk.drbd2080.out‚0î{«z2’- _wandbruntime‚4489  _runtime‚4489…;ÀòÉ“Ã trainer/global_step‚4499 % _timestamp‚1.7884325115522377e+09 _runtime‚4489.907247969 _step‚365 ' lr-AdamW/pg1‚0.00016179128066261974 ' lr-AdamW/pg2‚0.00016179128066261974mrÊ=Î”È ' lr-AdamW/pg1‚0.00016179128066261974 ' lr-AdamW/pg2‚0.00016179128066261974 trainer/global_step‚4499 % _timestamp‚1.7884325115522377e+09 _runtime‚4489.907247969 _step‚365íºÁ`2•- _wandbruntime‚4489  _runtime‚4489+¡ÀG–úO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618299997302272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 trainer/global_step‚4499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80281539 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618296865517056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚608697869958144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618296865517056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96681558 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35211201 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607965261844826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚366 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176939985 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9582781822298 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96681558 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚608697869958144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9602100952576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9598995558912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96658452 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚598382480022528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80281533 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9598995558912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35210649 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13331942523392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 _runtime‚4489.908169027 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618299997302272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9602100952576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80281539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47969015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533466631193600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9585885291222 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83179661 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83180216ÍÙƒ(—ÿO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80281533 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9598995558912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35210649 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13331942523392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 _runtime‚4489.908169027 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618299997302272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9602100952576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80281539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47969015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533466631193600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9585885291222 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83179661 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83180216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618299997302272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 trainer/global_step‚4499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80281539 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618296865517056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚608697869958144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618296865517056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96681558 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35211201 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607965261844826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚366 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176939985 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9582781822298 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96681558 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚608697869958144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9602100952576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9598995558912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96658452 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚598382480022528î©,2˜- _wandbruntime‚4490  _runtime‚4490!Ñ '™„N EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚608829364046336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884325124323287e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚618430471919104 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83197982 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83197444 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520247099534336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96702674 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚618433603717632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176978359 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚618433603717632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9604213279744 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176978359 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80298823 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533581980974080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9584890615398 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177001497 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47979265 _step‚367 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚608829390437888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚608829390437888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96679542 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚608829364046336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚598511902451712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚598511877023744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47979268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533582066206720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608099896536242 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520247127668736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13334938537984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9587994084530 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9604213279744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4499 _runtime‚4490.788052322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35218714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35218179 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80298823 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96679542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608096767639142 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302–Ã{Kã%š‰N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520247099534336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96702674 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚618433603717632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176978359 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚618433603717632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9604213279744 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176978359 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80298823 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533581980974080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9584890615398 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177001497 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47979265 _step‚367 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚608829390437888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚608829390437888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96679542 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚608829364046336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚598511902451712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚598511877023744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47979268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533582066206720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608099896536242 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520247127668736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13334938537984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9587994084530 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9604213279744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4499 _runtime‚4490.788052322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35218714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35218179 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80298823 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96679542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608096767639142 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚608829364046336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884325124323287e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚618430471919104 KBDeviceS“³¸,tatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83197982 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83197444ïlÞµS2›- _wandbruntime‚4490  _runtime‚4490NtcUœÏ 1loss/SRqqt4/assignment_loss‚1.867067115008831 0loss/FBt2/assignment_loss‚0.2984497954039398 _step‚368 /loss/SRqqt1/detection_loss‚1.17516143959673 1loss/SRqqt2/detection_loss‚2.6437043845653534 0loss/FBt4/assignment_loss‚0.1937974386115244 0loss/FBt3/assignment_loss‚0.2785101518301027 /loss/FBt3/detection_loss‚2.0814538544842174 &loss/total_loss‚44.425883759185126 % _timestamp‚1.7884325124345694e+09 2loss/SRqqt2/assignment_loss‚2.0561187836740697 /loss/FRt1/detection_loss‚1.9361002256924456 1loss/SRqqt4/detection_loss‚2.8059442436695097 trainer/global_step‚4499 _runtime‚4490.789538034 /loss/FBt2/detection_loss‚2.4309987910091877 .loss/FBt4/detection_loss‚1.503934570366428 /loss/FRt2/detection_loss‚2.4567601695293333 .loss/FBt1/assignment_loss‚0.07831095662921 /loss/FRt3/detection_loss‚2.1640486419200897 .loss/FRt4/detection_loss‚2.707965275725803 0loss/FRt1/assignment_loss‚1.7793157902630892 1loss/SRqqt3/detection_loss‚1.5500061595943613 /loss/FBt1/detection_loss‚0.7290615903615367 epoch‚3 /loss/FRt3/assignment_loss‚2.539588677639864 1loss/SRqqt3/assignment_loss‚1.232621955882076 /loss/FRt2/assignment_loss‚4.522936783185819 /loss/FRt4/assignment_loss‚4.112742366017522 2loss/SRqqt1/assignment_loss‚1.2812845985240795\ù6ZÔ /loss/FRt2/assignment_loss‚4.522936783185819 /loss/FRt4/assignment_loss‚4.112742366017522 2loss/SRqqt1/assignment_loss‚1.2812845985240795 1loss/SRqqt4/assignment_loss‚1.867067115008831 0loss/FBt2/assignment_loss‚0.2984497954039398 _step‚368 /loss/SRqqt1/detection_loss‚1.17516143959673 1loss/SRqqt2/detection_loss‚2.6437043845653534 0loss/FBt4/assignment_loss‚0.1937974386115244 0loss/FBt3/assignment_loss‚0.2785101518301027 /loss/FBt3/detection_loss‚2.0814538544842174 &loss/total_loss‚44.425883759185126 % _timestamp‚1.7884325124345694e+09 2loss/SRqqt2/assignment_loss‚2.0561187836740697 /loss/FRt1/detection_loss‚1.9361002256924456 1loss/SRqqt4/detection_loss‚2.8059442436695097 trainer/global_step‚4499 _runtime‚4490.789538034 /loss/FBt2/detection_loss‚2.4309987910091877 .loss/FBt4/detection_loss‚1.503934570366428 /loss/FRt2/detection_loss‚2.4567601695293333 .loss/FBt1/assignment_loss‚0.07831095662921 /loss/FRt3/detection_loss‚2.1640486419200897 .loss/FRt4/detection_loss‚2.707965275725803 0loss/FRt1/assignment_loss‚1.7793157902630892 1loss/SRqqt3/detection_loss‚1.5500061595943613 /loss/FBt1/detection_loss‚0.7290615903615367 epoch‚3 /loss/FRt3/assignment_loss‚2.539588677639864 1loss/SRqqt3/assignment_loss‚1.232621955882076ð{ì. œž‚(:‘ ‰¡åÔٿе gpu.0.gpu‚89.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚284.22' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.84400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ2£qŸ‚(:æ ‰¡åÔ®þú· network.recv‚ 1.62466e+06 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0$ disk./.usageGB‚278.4352912902832 disk.nvme0n1.in‚ 387.80859375' disk.nvme0c0n1.out‚34716.1005859375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚18.68602602126116 disk.rbd0.out‚0$ memory_percent‚94.04163360595703 proc.cpu.threads‚17 disk.rbd1.out‚2.453125 disk.sda1.in‚0 disk.drbd2080.out‚0% proc.memory.availableMB‚ 976.21875* proc.memory.percent‚18.003249168395996 cpu‚25.24148424340778 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 387.80859375 disk.sda.in‚0.203125 disk.drbd2080.in‚0% disk.nvme0n1.out‚34716.1005859375 disk.dm-1.out‚ 167.82421875 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2949.65234375 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 387.80859375" disk.dm-0.out‚34716.1005859375 disk.sda3.in‚0.203125 disk.sda3.out‚ 167.82421875 disk.drbd2266.out‚0 disk.sda.out‚ 167.82421875 network.sent‚ 1.0640266e+076€¥’ ‚(:‡ ˜¡åÔõ¥‡µ gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚299.435' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.887 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á?îz¡‚(:ï ˜¡åÔ¸¥ü¸ disk.rbd1.out‚2.453125 disk.sda3.in‚0.203125 disk.rbd0.in‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 387.80859375& disk.nvme0n1.out‚34718.44873046875 disk.drbd2266.out‚0 disk.sda1.in‚0$ memory_percent‚94.04032230377197* proc.memory.percent‚18.003535270690918 proc.cpu.threads‚17 network.sent‚ 1.074814e+07 disk.sda.out‚ 168.36328125 disk.drbd2265.in‚0 disk.sda1.out‚0 cpu‚25.240840146497874* disk./.usagePercent‚18.686080806873488 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚34718.44873046875 disk.dm-0.in‚ 387.80859375 disk.drbd1716.in‚0( proc.memory.availableMB‚ 976.43359375 network.recv‚ 1.630855e+06 disk.sda.in‚0.203125 disk.sda3.out‚ 168.36328125% disk./.usageGB‚278.43610763549805 disk.rbd0.out‚0 disk.nvme0n1.in‚ 387.80859375( disk.nvme0c0n1.out‚34718.44873046875 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2949.69921875 disk.dm-1.out‚ 168.36328125 disk.drbd2266.in‚0ìhµdž¢‚(:“ §¡åÔñªÎµ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚288.091' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.618199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«•…žk£‚(:à §¡åÔå×Ô· disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2949.69921875* disk./.usagePercent‚18.686104871581705 disk.dm-1.out‚169.0625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.out‚169.0625 disk.drbd1716.in‚0$ memory_percent‚94.04053688049316# disk.dm-0.out‚34719.26513671875 disk.drbd2265.out‚0' proc.memory.availableMB‚ 976.3984375 cpu‚25.240171905510365 proc.cpu.threads‚17 network.sent‚ 1.0754208e+07 network.recv‚ 1.63523e+06 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚2.453125 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda3.in‚0.203125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 387.80859375 disk.sda.out‚169.0625# disk./.usageGB‚278.436466217041 disk.nvme0n1.in‚ 387.80859375 disk.sda2.in‚0( disk.nvme0c0n1.out‚34719.26513671875* proc.memory.percent‚18.003535270690918& disk.nvme0n1.out‚34719.26513671875 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 387.80859375ÙÆúÚ2¤- _wandbruntime‚4536  _runtime‚4536Rkwȥ $ _timestamp‚1.788432558008798e+09 _runtime‚4536.363873142 _step‚369 ' lr-AdamW/pg1‚0.00016058411726993228 ' lr-AdamW/pg2‚0.00016058411726993228 trainer/global_step‚4549-êokÍ¦Ç $ _timestamp‚1.788432558008798e+09 _runtime‚4536.363873142 _step‚369 ' lr-AdamW/pg1‚0.00016058411726993228 ' lr-AdamW/pg2‚0.00016058411726993228 trainer/global_step‚4549ñ­±c2§- _wandbruntime‚4536  _runtime‚4536 Ç}Ö(¨úO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48481662 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚604852355511296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9707731059200 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _step‚370 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚525755260011008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178862037 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624977200033280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81145739 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614540565151834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615272574367744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4536.364821751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13478424017920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚4549 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97739410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9688235068506 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81145739 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615272600759296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81145733 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35591874 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539233684028928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984æçr(©ÿO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚604852355511296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9707731059200 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _step‚370 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚525755260011008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178862037 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624977200033280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81145739 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614540565151834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615272574367744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4536.364821751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13478424017920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚4549 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97739410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9688235068506 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81145739 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615272600759296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81145733 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35591874 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539233684028928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48481662òÿNOü2ª- _wandbruntime‚4537  _runtime‚4537[Wì #«ƒN \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9693447330738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35599539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _runtime‚4537.243268242 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97737394 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚625113938233856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9706737979392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178923549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84090947 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚525867642740736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97737394 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚371 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539349023732224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97760526 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚604981752512512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35599032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178900411 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9690343861606 trainer/global_step‚4549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615404068455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81163023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178900411 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525867670875136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚625113938233856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615404094847488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚604981727084544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614672070946150 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81163017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 $ _timestamp‚1.788432558887784e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚625113938233856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615404094847488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚604981727084544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614672070946150 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81163017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 $ _timestamp‚1.788432558887784e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _runtime‚4537.243268242 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97737394 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚625113938233856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9706737979392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178923549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84090947 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚525867642740736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97737394 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚371 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539349023732224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97760526 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚604981752512512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35599032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178900411 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9690343861606 trainer/global_step‚4549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615404068455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81163023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178900411 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525867670875136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚631657534549504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621847305168896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545000602352640 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9810255772160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚611322205572096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚631660666334720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82009933 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13624827269632 % _timestamp‚1.7884326042613802e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531375775083008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621847305168896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚611322180144128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13624882284032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82009939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98797262 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _runtime‚4582.616574273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_/™q*^start/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9796791783638 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9813361165824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48994312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9810255772160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚531375803217408 trainer/global_step‚4599 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545000685501440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚621115868458842 _step‚374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98797262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621847278777344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚631657534549504+n—N(»ÿO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9810255772160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚611322205572096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚631660666334720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82009933 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13624827269632 % _timestamp‚1.7884326042613802e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531375775083008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621847305168896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚611322180144128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13624882284032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82009939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98797262 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _runtime‚4582.616574273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9796791783638 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9813361165824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48994312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9810255772160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚531375803217408 trainer/global_step‚4599 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545000685501440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚621115868458842 _step‚374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98797262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621847278777344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚631657534549504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82009939 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚631657534549504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621847305168896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545000602352640 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878ötZ2¼- _wandbruntime‚4583  _runtime‚4583W ~± '½„N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545116022716928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621978772865536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚4583.497617488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180845601 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84982490 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82027217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚611451577145344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98818378 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98795246 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180822463 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621978772865536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49004565 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9812368086016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚621247374253158 _step‚375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98795246 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚631794272750080 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180845601 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚631791140951552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9815473492992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82027223 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13627808635392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84982490 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82027217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚611451577145344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98818378 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98795246 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180822463 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621978772865536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49004565 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9812368086016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚621247374253158 _step‚375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98795246 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚631794272750080 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180845601 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚631791140951552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9815473492992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82027223 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13627808635392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚4583.497617488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180845601 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚536996346423808 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13771528875008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚638337869065728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49506965 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85857511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638337869065728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36351078 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚378 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13771473860608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 % _timestamp‚1.7884326508130214e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638341000850944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9918991272448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚627691171765850 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82874139 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9918991272448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9902245029846 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82874133 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550767792150016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚628422009578496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚617792030204928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99832008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9915885878784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚628421983186944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚617792055632896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99855114 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99832008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚550767875298816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4649 _runtime‚4629.168289747 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0eï«(ÍÿO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82874133 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550767792150016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚628422009578496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚617792030204928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99832008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9915885878784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚628421983186944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚617792055632896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99855114 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99832008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚550767875298816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4649 _runtime‚4629.168289747 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82874139 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85858043 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚536996346423808 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13771528875008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚638337869065728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49506965 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85857511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638337869065728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36351078 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚378 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13771473860608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 % _timestamp‚1.7884326508130214e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638341000850944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9918991272448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚627691171765850 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82874139 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9918991272448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9902245029846 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878úÕÀ 2Î- _wandbruntime‚4630  _runtime‚4630ž´tþvÏ„N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99876230 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182744515 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537108757287936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚617921452634112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9904353823154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49517218 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36358672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85875890 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49517215 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182767653 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85875388 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚638471475467776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚628553477275136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚628553503666688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚617921427206144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚628553477275136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13774406749696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82891417 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537108729153536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13774461750784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚4649 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9921103599616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚638474607266304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9917998192640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82891417 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99853098 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚628553503666688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99853098 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 _step‚379 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9901250354022 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 _runtime‚4630.050490927 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9917998192640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9921103599616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚550883219038720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627822677560166 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884326516945505e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182767653 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82891423 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180a$9#'ЉN REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82891423 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99876230 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182744515 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537108757287936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚617921452634112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9904353823154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49517218 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36358672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85875890 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49517215 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182767653 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85875388 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚638471475467776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚628553477275136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚628553503666688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚617921427206144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚628553477275136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13774406749696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82891417 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537108729153536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13774461750784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚4649 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9921103599616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚638474607266304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9917998192640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82891417 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99853098 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚628553503666688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99853098 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 _step‚379 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9901250354022 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 _runtime‚4630.050490927 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9917998192640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9921103599616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚550883219038720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627822677560166 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884326516945505e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182767653ûMmò.2Ñ- _wandbruntime‚4630  _runtime‚4630#+ýV\ÒÖ .loss/FBt4/detection_loss‚1.830865136285623 2loss/SRqqt3/assignment_loss‚1.2152095530706812 1loss/FBt1/assignment_loss‚0.04158723502576843 /loss/FRt3/detection_loss‚2.6414970360696315 1loss/SRqqt4/detection_loss‚2.5845999121665955 2loss/SRqqt4/assignment_loss‚1.3245781750107806 1loss/SRqqt3/detection_loss‚1.4131623827130861 .loss/FBt2/detection_loss‚2.470982687039809 0loss/FRt4/assignment_loss‚3.7668687462806703 .loss/FRt4/detection_loss‚3.028278772830963 0loss/SRqqt2/detection_loss‚2.640198915235458 trainer/global_step‚4649 2loss/SRqqt1/assignment_loss‚1.1622921270459439 _step‚380 2loss/SRqqt2/assignment_loss‚1.6831483812101427 1loss/FBt4/assignment_loss‚0.10976729873018178 .loss/FRt2/detection_loss‚2.618818720181783 epoch‚3 % _timestamp‚1.7884326516965802e+09 /loss/FRt1/assignment_loss‚2.137130146295252 0loss/FRt2/assignment_loss‚4.3794594526290895 0loss/FRt3/assignment_loss‚2.6588620126247404 /loss/FRt1/detection_loss‚1.6242579032837505 %loss/total_loss‚44.27680189124638 _runtime‚4630.051522692 0loss/FBt2/assignment_loss‚0.3041412571275776 1loss/FBt3/assignment_loss‚0.14402854383330455 1loss/SRqqt1/detection_loss‚1.2244890395439032 /loss/FBt1/detection_loss‚0.7748367523742934 /loss/FBt3/detection_loss‚2.4977417046373542GdŠaÓÛ .loss/FBt4/detection_loss‚1.830865136285623 2loss/SRqqt3/assignment_loss‚1.2152095530706812 1loss/FBt1/assignment_loss‚0.04158723502576843 /loss/FRt3/detection_loss‚2.6414970360696315 1loss/SRqqt4/detection_loss‚2.5845999121665955 2loss/SRqqt4/assignment_loss‚1.3245781750107806 1loss/SRqqt3/detection_loss‚1.4131623827130861 .loss/FBt2/detection_loss‚2.470982687039809 0loss/FRt4/assignment_loss‚3.7668687462806703 .loss/FRt4/detection_loss‚3.028278772830963 0loss/SRqqt2/detection_loss‚2.640198915235458 trainer/global_step‚4649 2loss/SRqqt1/assignment_loss‚1.1622921270459439 _step‚380 2loss/SRqqt2/assignment_loss‚1.6831483812101427 1loss/FBt4/assignment_loss‚0.10976729873018178 .loss/FRt2/detection_loss‚2.618818720181783 epoch‚3 % _timestamp‚1.7884326516965802e+09 /loss/FRt1/assignment_loss‚2.137130146295252 0loss/FRt2/assignment_loss‚4.3794594526290895 0loss/FRt3/assignment_loss‚2.6588620126247404 /loss/FRt1/detection_loss‚1.6242579032837505 %loss/total_loss‚44.27680189124638 _runtime‚4630.051522692 0loss/FBt2/assignment_loss‚0.3041412571275776 1loss/FBt3/assignment_loss‚0.14402854383330455 1loss/SRqqt1/detection_loss‚1.2244890395439032 /loss/FBt1/detection_loss‚0.7748367523742934 /loss/FBt3/detection_loss‚2.4977417046373542üŒ!¹‘Ô‚(:† ¢åÔП¶ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚293.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08 ÚtÕ‚(:é ¢åÔºÔ…· network.recv‚ 1.669483e+06) disk./.usagePercent‚18.68653189214882$ disk./.usageGB‚278.4428291320801 disk.sda.out‚ 172.8359375 disk.nvme0n1.in‚ 387.80859375 disk.sda2.out‚0 disk.dm-1.out‚ 172.8359375 disk.rbd1.out‚2.453125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.203125& disk.nvme0n1.out‚34731.14697265625$ memory_percent‚94.04511451721191' proc.memory.availableMB‚ 975.6484375 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0# disk.dm-0.out‚34731.14697265625 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2949.84765625" disk.nvme0c0n1.in‚ 387.80859375 network.sent‚ 1.0999598e+07 disk.rbd0.out‚0 disk.sda3.out‚ 172.8359375* proc.memory.percent‚18.004441261291504 disk.drbd1716.out‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚34731.14697265625 disk.dm-0.in‚ 387.80859375 cpu‚25.23573367212378¨—ؤœÖ‚(:‘ Ÿ¢åÔ×æµ gpu.0.gpu‚52.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚130.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.139999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¶_çwׂ(:ì Ÿ¢åÔèðÒ· disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 387.80859375" disk.nvme0c0n1.in‚ 387.80859375 disk.dm-0.in‚ 387.80859375 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 975.65234375 disk.drbd2266.in‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚34732.10791015625 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125" proc.memory.rssMB‚ 2949.8984375 disk.sda3.out‚ 173.66796875 disk.sda1.in‚0 disk.sda.out‚ 173.66796875* proc.memory.percent‚18.004751205444336% disk./.usageGB‚278.44324493408203 disk.rbd1.in‚0 disk.rbd1.out‚2.453125# disk.dm-0.out‚34732.10791015625 disk.dm-1.out‚ 173.66796875 cpu‚25.235077571042865 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚34732.10791015625 disk.dm-1.in‚0.203125 proc.cpu.threads‚17) disk./.usagePercent‚18.68655979697005 disk.sda2.out‚0" memory_percent‚94.045090675354 network.sent‚ 1.1108376e+07 network.recv‚ 1.676954e+06 disk.drbd1716.in‚0 P÷’Ø‚(:‡ ®¢åÔ–¢Žµ gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚222.995' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.599 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r—“ tÙ‚(:é ®¢åԾ¤· network.recv‚ 1.680924e+06% disk.nvme0n1.out‚34734.3720703125" disk.dm-0.out‚34734.3720703125# proc.memory.rssMB‚ 2949.94921875 disk.rbd1.in‚0 disk.sda.out‚ 174.05859375 disk.drbd2266.out‚0 disk.rbd1.out‚2.453125 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 174.05859375* proc.memory.percent‚18.005061149597168 cpu‚25.234436513486116 network.sent‚ 1.1114194e+07% disk./.usageGB‚278.44405364990234 disk.sda3.in‚0.203125 disk.sda3.out‚ 174.05859375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 387.80859375 disk.dm-0.in‚ 387.80859375 disk.sda1.in‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.drbd2266.in‚0$ memory_percent‚94.04582977294922% proc.memory.availableMB‚ 975.53125) disk./.usagePercent‚18.68661407056731' disk.nvme0c0n1.out‚34734.3720703125 disk.sda1.out‚0 disk.nvme0n1.in‚ 387.80859375 proc.cpu.threads‚17’ùè2Ú- _wandbruntime‚4675  _runtime‚4675¬ ÆÍÈÛ ' lr-AdamW/pg1‚0.00015689074955743063 ' lr-AdamW/pg2‚0.00015689074955743063 trainer/global_step‚4699 $ _timestamp‚1.788432697280683e+09 _runtime‚4675.635747701 _step‚381äÉÍÜÇ $ _timestamp‚1.788432697280683e+09 _runtime‚4675.635747701 _step‚381 ' lr-AdamW/pg1‚0.00015689074955743063 ' lr-AdamW/pg2‚0.00015689074955743063 trainer/global_step‚4699ýóS“ò2Ý- _wandbruntime‚4675  _runtime‚4675¤ Yç%Þ„P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634996687596544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86752464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13917907722752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100912966 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86751901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645021335367168 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚4699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50019612 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556534797352960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10021515985408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645018203581952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 % _timestamp‚1.7884326972815177e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50019615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36732289 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83738339 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634996713988096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10004594807130 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _runtime‚4675.636730555 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83738333 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100912966 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚634266475072858 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542616889630208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10007698276054 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645018203581952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚634996687596544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10024621379072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10024621379072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13917850611200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚382 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556534712107008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚624261880265728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180¢ZXï(߉P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚4699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50019612 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556534797352960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10021515985408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645018203581952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 % _timestamp‚1.7884326972815177e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50019615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36732289 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83738339 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634996713988096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10004594807130 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _runtime‚4675.636730555 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83738333 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100912966 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚634266475072858 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542616889630208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10007698276054 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645018203581952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚634996687596544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10024621379072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10024621379072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13917850611200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚382 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556534712107008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚624261880265728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634996687596544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86752464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13917907722752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100912966 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86751901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645021335367168þÇÙÎÈ2à- _wandbruntime‚4676  _runtime‚4676Ä©ì“'áN C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83755623 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184689705 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚556650142898176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4699 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645154941782528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184666567 _runtime‚4676.514534853 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10009807069362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚556650057665536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚634401109764274 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36740395 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83755617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542729272359936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100910950 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624391277266944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10006703600230 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 $ _timestamp‚1.788432698158784e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83755617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚383 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645154941782528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635128208076288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83755623 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10026733706240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50029865 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚634397980867174 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184689705 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86769721 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚624391302694912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635128181684736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100934082 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100910950 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13920842403840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10026733706240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13920785305600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10023628299264 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36740395 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83755617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542729272359936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100910950 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624391277266944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10006703600230 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 $ _timestamp‚1.788432698158784e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83755617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚383 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645154941782528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635128208076288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83755623 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10026733706240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50029865 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚634397980867174 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184689705 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86769721 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚624391302694912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635128181684736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100934082 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100910950 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13920842403840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10026733706240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13920785305600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10023628299264 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4699 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645154941782528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÿ³®W‚2ã- _wandbruntime‚4676  _runtime‚4676Vó¥VäÐ /loss/FRt3/detection_loss‚2.0217546762691603 0loss/SRqqt1/detection_loss‚1.141162062126599 0loss/SRqqt4/detection_loss‚2.540822702859129 1loss/SRqqt2/assignment_loss‚2.311364288840975 /loss/FBt1/detection_loss‚0.9736672218534208 /loss/FRt1/detection_loss‚1.8231538243469645 .loss/FRt2/detection_loss‚3.108654120029547 0loss/SRqqt2/detection_loss‚2.549776222024645 1loss/SRqqt3/detection_loss‚1.3700775247934747 0loss/FRt1/assignment_loss‚1.7468701874623533 1loss/FBt4/assignment_loss‚0.14193627973043815 /loss/FRt4/detection_loss‚3.3355036675930023 /loss/FBt2/detection_loss‚2.9171308763325214 epoch‚3 % _timestamp‚1.7884326981606457e+09 1loss/SRqqt3/assignment_loss‚1.400841920164554 0loss/FBt3/assignment_loss‚0.4564689930800038 .loss/FBt4/detection_loss‚1.143140028295822 trainer/global_step‚4699 _step‚384 0loss/FRt2/assignment_loss‚3.9525032532520785 0loss/FRt3/assignment_loss‚2.2718236302887953 0loss/FBt2/assignment_loss‚0.3771192613057224 _runtime‚4676.515928521 .loss/FRt4/assignment_loss‚3.50343571305275 2loss/SRqqt1/assignment_loss‚1.1074026258808842 1loss/FBt1/assignment_loss‚0.03692130655149467 $loss/total_loss‚45.3051489281234 2loss/SRqqt4/assignment_loss‚1.9037448540329933 .loss/FBt3/detection_loss‚3.169873687956068%ëé±åÕ .loss/FRt2/detection_loss‚3.108654120029547 0loss/SRqqt2/detection_loss‚2.549776222024645 1loss/SRqqt3/detection_loss‚1.3700775247934747 0loss/FRt1/assignment_loss‚1.7468701874623533 1loss/FBt4/assignment_loss‚0.14193627973043815 /loss/FRt4/detection_loss‚3.3355036675930023 /loss/FBt2/detection_loss‚2.9171308763325214 epoch‚3 % _timestamp‚1.7884326981606457e+09 1loss/SRqqt3/assignment_DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562301631024128 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84602533 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101970818 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640844907276758 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562301714172928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14064281336320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50532262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚651698538098176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚641571392006144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101947712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10110048053338 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101970818 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10130251485696 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚641571418397696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87645015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37112219 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548237404702208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _runtime‚ 4721.92210802 _step‚386 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186550245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚641571392006144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚640841778379866 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10127146092032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚651701669883392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10127146092032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚651698538098176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚641571418397696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186573357 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84602533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50532265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630731730326528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14064226321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37112750 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101947712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84602539 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84602539 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚548237432836608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚651701669883392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10130251485696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562301631024128 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84602533 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101970818 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640844907276758 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562301714172928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14064281336320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50532262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚651698538098176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚641571392006144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101947712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10110048053338 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101970818 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10130251485696 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚641571418397696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87645015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37112219 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548237404702208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _runtime‚ 4721.92210802 _step‚386 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186550245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚641571392006144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚640841778379866 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10127146092032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚651701669883392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10127146092032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚651698538098176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚641571418397696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186573357 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84602533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50532265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630731730326528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14064226321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37112750 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101947712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84602539 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84602539 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚548237432836608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531‚ Ž:2ò- _wandbruntime‚4722  _runtime‚4722®Œjú‹óN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87662436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37120446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101991934 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10129258405888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186611757 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630861152755712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50542515 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630861127327744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚651835276298752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84619817 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101968802 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚548349843700736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10129258405888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10132363812864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚4749 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚562416973794816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186588619 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640976413071282 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101991934 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14067158228480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚387 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562417059027456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4722.799599527 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440tõd'ô’N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10112156846438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37119921 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10115260315570 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186611757 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚641702912485888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚640973284174182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87662436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37120446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101991934 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10129258405888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186611757 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630861152755712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50542515 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630861127327744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚651835276298752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84619817 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101968802 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚548349843700736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10129258405888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10132363812864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚4749 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚562416973794816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186588619 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640976413071282 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101991934 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14067158228480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚387 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562417059027456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4722.799599527 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚4799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188472297 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103028670 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103005564 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10235881592320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85466733 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553857947908608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14210784501760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 _step‚390 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188495409 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10232776198656 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658378872614400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658382004399616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568068815559168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10218604768470 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10235881592320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884327900338655e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032j·àƒ‰P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37494333 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _runtime‚4768.389134319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/øº’¹ý%global_step‚4799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188472297 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103028670 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103005564 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10235881592320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85466733 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553857947908608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14210784501760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 _step‚390 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188495409 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10232776198656 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658378872614400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658382004399616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568068815559168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10218604768470 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10235881592320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884327900338655e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88538697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51044915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚648146096415744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85466739 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚648146122807296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103028670 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚648146096415744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚648146122807296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37493785 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553857976043008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚647420210583766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51044912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10232776198656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14210839516160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85466733 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552†鿇2„- _wandbruntime‚4769  _runtime‚4769[¦dç'…N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188510671 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚648277590503936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚648277590503936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51055168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647548587481190 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188533809 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚568184083891712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10234888512512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568184167027200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚391 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10220713561778 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚648277616895488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103049786 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10234888512512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553970386907136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 $ _timestamp‚1.788432790916332e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10237993919488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637330977388544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85484023 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88556697 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10217610092646 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37501532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88557224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658515610814976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85484017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51055165 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚4799³_s±'†’N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚4799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188510671 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚648277590503936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚648277590503936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51055168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647548587481190 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188533809 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚568184083891712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10234888512512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568184167027200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚391 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10220713561778 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚648277616895488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103049786 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10234888512512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553970386907136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 $ _timestamp‚1.788432790916332e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10237993919488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637330977388544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85484023 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88556697 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10217610092646 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37501532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88557224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658515610814976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85484017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51055165‡ê&Í2‡- _wandbruntime‚4769  _runtime‚4769Q&B\ˆÖ 0loss/FRt1/assignment_loss‚1.7659033238887787 epoch‚3 _step‚392 2loss/SRqqt3/assignment_loss‚1.1907773025763722 0loss/FBt2/assignment_loss‚0.3447055958888712 /loss/FBt2/detection_loss‚1.8148586614565416 2loss/SRqqt1/assignment_loss‚1.2428266276191506 0loss/FBt3/assignment_loss‚0.2766947177151451 1loss/SRqqt1/detection_loss‚1.1782600120641291 1loss/SRqqt3/detection_loss‚1.2938915165380025 0loss/SRqqt4/detection_loss‚2.559814304113388 0loss/FRt2/assignment_loss‚3.5578490631146864 /loss/FRt4/assignment_loss‚4.036220186097282 .loss/FBt3/detection_loss‚3.940861761569977 % _timestamp‚1.7884327909180489e+09 _runtime‚4769.273008147 0loss/FRt3/assignment_loss‚2.5780932487181896 2loss/SRqqt2/assignment_loss‚2.0124770432062773 2loss/FBt1/assignment_loss‚0.021713752628425377 .loss/FRt1/detection_loss‚1.640705929888833 /loss/FBt1/detection_loss‚0.7558696729705688 .loss/FBt4/detection_loss‚2.278701432404064 trainer/global_step‚4799 1loss/FBt4/assignment_loss‚0.38923896921763285 /loss/FRt4/detection_loss‚2.8273621116365706 %loss/total_loss‚45.69916362840398 2loss/SRqqt4/assignment_loss‚2.3158909641206264 /loss/FRt2/detection_loss‚2.5950613387606363 .loss/FRt3/detection_loss‚2.104535662902976 1loss/SRqqt2/detection_loss‚2.9768504293068596­Ña‰Û 2loss/SRqqt4/assignment_loss‚2.3158909641206264 /loss/FRt2/detection_loss‚2.5950613387606363 .loss/FRt3/detection_loss‚2.104535662902976 1loss/SRqqt2/detection_loss‚2.9768504293068596 0loss/FRt1/assignment_loss‚1.7659033238887787 epoch‚3 _step‚392 2loss/SRqqt3/assignment_loss‚1.1907773025763722 0loss/FBt2/assignment_loss‚0.3447055958888712 /loss/FBt2/detection_loss‚1.8148586614565416 2loss/SRqqt1/assignment_loss‚1.2428266276191506 0loss/FBt3/assignment_loss‚0.2766947177151451 1loss/SRqqt1/detection_loss‚1.1782600120641291 1loss/SRqqt3/detection_loss‚1.2938915165380025 0loss/SRqqt4/detection_loss‚2.559814304113388 0loss/FRt2/assignment_loss‚3.5578490631146864 /loss/FRt4/assignment_loss‚4.036220186097282 .loss/FBt3/detection_loss‚3.940861761569977 % _timestamp‚1.7884327909180489e+09 _runtime‚4769.273008147 0loss/FRt3/assignment_loss‚2.5780932487181896 2loss/SRqqt2/assignment_loss‚2.0124770432062773 2loss/FBt1/assignment_loss‚0.021713752628425377 .loss/FRt1/detection_loss‚1.640705929888833 /loss/FBt1/detection_loss‚0.7558696729705688 .loss/FBt4/detection_loss‚2.278701432404064 trainer/global_step‚4799 1loss/FBt4/assignment_loss‚0.38923896921763285 /loss/FRt4/detection_loss‚2.8273621116365706 %loss/total_loss‚45.69916362840398ˆ¦,fÕŠ‚(:“ —£åÔŽ»¶µ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memory<È×½AllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚255.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oî¾]‹‚(:Ò —£åÔÏ㡸 network.recv‚ 1.716235e+06 disk.rbd1.out‚2.453125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125" disk.nvme0c0n1.in‚ 387.80859375# disk.nvme0c0n1.out‚ 34745.484375 disk.dm-0.in‚ 387.80859375 disk.sda3.in‚0.203125 proc.memory.rssMB‚2950& proc.memory.percent‚18.00537109375 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 975.37890625 cpu‚25.230283508242323$ disk./.usageGB‚278.4488754272461 disk.sda1.in‚0 disk.sda.out‚ 177.69140625 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 387.80859375 disk.dm-1.in‚0.203125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 177.69140625 disk.rbd1.in‚0! disk.nvme0n1.out‚ 34745.484375) disk./.usagePercent‚18.68693766409059 disk.drbd2266.out‚0 disk.dm-0.out‚ 34745.484375 disk.sda3.out‚ 177.69140625 disk.rbd0.in‚0$ memory_percent‚94.04675960540771 network.sent‚ 1.1360867e+07 disk.drbd2265.in‚0‚ë䌂(:’ ¦£åÔùÃÿµ gpu.0.gpu‚76.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚256.234' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.24679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆº‘w‚(:ì ¦£åÔÉż¸ disk.rbd1.out‚2.453125# disk.dm-0.out‚34747.28759765625 disk.sda3.in‚0.203125 disk.sda1.in‚0 network.sent‚ 1.1469464e+07 network.recv‚ 1.723245e+06* disk./.usagePercent‚18.686990401642642$ disk./.usageGB‚278.4496612548828( disk.nvme0c0n1.out‚34747.28759765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0$ memory_percent‚94.04723644256592 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 cpu‚25.229719091037616 disk.drbd2265.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.005681037902832 disk.sda.in‚0.203125& disk.nvme0n1.out‚34747.28759765625" disk.nvme0c0n1.in‚ 387.80859375 disk.dm-0.in‚ 387.80859375 disk.sda3.out‚ 178.2109375 disk.drbd1716.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2950.05078125 disk.drbd1716.out‚0 disk.dm-1.out‚ 178.2109375( proc.memory.availableMB‚ 975.30078125 disk.drbd2080.out‚0 disk.sda.out‚ 178.2109375 disk.nvme0n1.in‚ 387.80859375 disk.sda2.in‚0‰n«Ô“Ž‚(:ˆ µ£åÔݵŵ gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚167.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýýâ~w‚(:ì µ£åÔ¢²Ž· disk.nvme0n1.in‚ 387.80859375 disk.sda.out‚ 179.01171875' proc.memory.availableMB‚ 974.8828125# proc.memory.rssMB‚ 2950.19921875& disk.nvme0n1.out‚34748.11572265625 disk.sda2.in‚0 disk.dm-1.out‚ 179.01171875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 cpu‚25.22912337300457 disk.rbd1.out‚2.453125 disk.sda3.out‚ 179.01171875 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 network.recv‚ 1.728366e+06% disk./.usageGB‚278.45008850097656 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚18.68701907448648 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 387.80859375# disk.dm-0.out‚34748.11572265625 disk.sda.in‚0.203125 disk.drbd2265.out‚0# memory_percent‚94.0497875213623 disk.dm-0.in‚ 387.80859375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚18.006587028503418 network.sent‚ 1.1475784e+07( disk.nvme0c0n1.out‚34748.11572265625 disk.rbd0.in‚0‚3–”’‚(:‡ Ä£åÔυʵ gpu.0.gpu‚35.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚123.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê<m‘‚(:â Ä£åÔîÒ–· disk.sda3.in‚0.203125 disk.sda1.out‚0$ memory_percent‚93.60935688018799 disk.dm-1.out‚ 179.51953125 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 387.8203125 disk.dm-0.out‚34752.23046875 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1047.04296875$ disk./.usageGB‚278.4497947692871 disk.rbd1.out‚2.453125 disk.sda3.out‚ 179.51953125 disk.drbd2080.out‚0 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2878.36328125 disk.sda.out‚ 179.51953125 network.sent‚ 1.1481114e+07 network.recv‚ 1.731717e+06# disk.nvme0n1.out‚34752.23046875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 cpu‚25.23977457605251 disk.dm-1.in‚0.203125% disk.nvme0c0n1.out‚34752.23046875 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 387.8203125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.568135261535645) disk./.usagePercent‚18.68699936190634 disk.sda2.out‚0 disk.dm-0.in‚ 387.8203125˜ìžùž’‚(:“ Ó£åÔóʹµ gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚298.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¡¯Ðq“‚(:æ Ó£åÔõàÁ· disk.sda.out‚ 180.203125 disk.sda3.out‚ 180.203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 405.15234375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34792.48095703125! proc.memory.rssMB‚ 2950.890625* proc.memory.percent‚18.010807037353516 network.recv‚ 1.736421e+06% disk./.usageGB‚278.48181915283203 disk.sda2.in‚0# disk.dm-0.out‚34792.48095703125$ memory_percent‚94.05291080474854 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚34792.48095703125 disk.dm-1.in‚0.203125( proc.memory.availableMB‚ 974.37109375 cpu‚25.25169562636339 network.sent‚ 1.1487135e+07) disk./.usagePercent‚18.68914854515629 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 180.203125 disk.sda.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 405.15234375 disk.rbd1.out‚2.453125" disk.nvme0c0n1.in‚ 405.15234375 disk.sda3.in‚0.203125 proc.cpu.threads‚17W¿¦”‚(:„ â£åÔÃÑ­µ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™OÔs•‚(:è â£åÔ·«®· disk.drbd2265.out‚0 disk.dm-1.out‚ 182.1796875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚2.453125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.203125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 436.23828125! disk.dm-0.out‚34871.216796875 disk.sda1.out‚0$ memory_percent‚93.61417293548584* proc.memory.percent‚17.571425437927246 cpu‚25.264421149136663 disk.nvme0n1.in‚ 436.23828125 disk.dm-1.in‚0.203125% disk./.usageGB‚278.55655670166016 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚34871.216796875 disk.sda3.in‚0.203125 disk.sda3.out‚ 182.1796875) proc.memory.availableMB‚ 1046.25390625# proc.memory.rssMB‚ 2878.90234375 network.sent‚ 1.1493095e+07 network.recv‚ 1.740658e+06* disk./.usagePercent‚18.694164244767126 disk.sda.out‚ 182.1796875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚34871.216796875 disk.dm-0.in‚ 436.23828125Îñt£2–- _wandbruntime‚4857  _runtime‚4857¿qܬs*—íT MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚51897812 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚667739904392704 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 120738304 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚86752944 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690840 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 % _timestamp‚1.7884328789919937e+09 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 190921748 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚86752949 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10338462442496 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 104187082 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚51897810 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚10336132835840 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚646151825272832 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353901056 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 104168804 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚667737550491648 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚14344188628480 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18278 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚126 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚656472804866366 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚646151801942016 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 190940031 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14344291718144 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10338462442496 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚4857.347612807 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚576049908532736 trainer/global_step‚4835 _step‚393 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 190940031 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 190921748 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 104168804 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚128 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚657401417655808 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10336132835840 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚576050029271040 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚667737550491648 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚657401441950208 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27964 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚561705719904256 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚656470453844710 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 104187082 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚561705737552896 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10318651902694 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚86752949 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚86752944 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103089664 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚657401441950208 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚89904052 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3277969920 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚657401417655808 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656Ï £x*˜òT aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10338462442496 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚4857.347612807 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚576049908532736 trainer/global_step‚4835 _step‚393 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 190940031 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 190921748 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 104168804 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚128 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚657401417655808 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10336132835840 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚576050029271040 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚667737550491648 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚657401441950208 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27964 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚561705719904256 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚656470453844710 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 104187082 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚561705737552896 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10318651902694 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚86752949 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚86752944 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103089664 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚657401441950208 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚89904052 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3277969920 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚657401417655808 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1697 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚51897812 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚667739904392704 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 120738304 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚86752944 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690840 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 % _timestamp‚1.7884328789919937e+09 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 190921748 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚86752949 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10338462442496 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 104187082 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚51897810 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚10336132835840 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚646151825272832 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353901056 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 104168804 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚667737550491648 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚14344188628480 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18278 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚126 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚656472804866366 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚646151801942016 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 190940031 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14344291718144‰›0Ùô2™- _wandbruntime‚4858  _runtime‚4858¦NªIšöR _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚657491010265600 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 104176498 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10336630853120 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚667829995019776 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚667827641118720 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚86766147 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚657491034560000 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚51907474 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606656 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 190942640 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚89918462 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3277969920 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1697 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18283 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚576125804013568 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚38010986 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021656 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚646236751542272 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚86766142 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚656558250918270 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚561780996629504 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1878 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 190942640 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚657491010265600 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10338960459776 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103089664 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14344686645760 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚646236774873088 $ _timestamp‚1.788432880072548e+09 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10319148354342 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚576125683275264 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10321476045182 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚657491034560000 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚86766142 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚89918334 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚656555899896614 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚968 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 104176498 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚128 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚561781014278144 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚10336630853120 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0t¤‘)›ûR E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2353901056 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 28510781440 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚667829995019776 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚667827641118720 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚86766147 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚657491034560000 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚51907474 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606656 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 190942640 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚89918462 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3277969920 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1697 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18283 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚576125804013568 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚38010986 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021656 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚646236751542272 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚86766142 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚656558250918270 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚561780996629504 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1878 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 190942640 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚657491010265600 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10338960459776 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103089664 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14344686645760 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚646236774873088 $ _timestamp‚1.788432880072548e+09 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10319148354342 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚576125683275264 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10321476045182 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚657491034560000 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚86766142 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚89918334 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚656555899896614 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚968 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 104176498 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚128 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚561781014278144 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚10336630853120 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚657491010265600 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 104176498 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10336630853120 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956322312355042 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5321015119552612 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 _runtime‚4872.789952214 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988184571266174 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 particle/accuracy_7_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4317580759525299 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178707957267761 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4112611413002014 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5304415822029114 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.07575757801532745 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1997126340866089 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3373550474643707 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5337821841239929 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34259262681007385 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4599137604236603 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2613636255264282 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17637430131435394 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8920972943305969 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08517809212207794 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.20882493257522583 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04317142814397812 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8434349298477173 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.442631334066391 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 +jet/accuracy_2_of_2‚0.09015357494354248 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.423658162355423 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010552243329584599 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4748994708061218 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.003138292348012328 particle/accuracy_2_of_2‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17958088219165802 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.452562540769577 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5896391272544861 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01725367270410061 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.3227407932281494 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4300000071525574 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5147100687026978 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5961538553237915 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.30435436964035034 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12997233867645264 particle/accuracy_5_of_5‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21435414254665375 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33731013536453247 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31349167227745056 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 trainer/global_step‚4835 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15859845280647278 <"Purity/0FRt1SRqqt2FBt/event_purity‚0.0024630543775856495 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4262963831424713 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.048190534114837646 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2689529359340668 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915844798088074 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4379558265209198 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982104480266571 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4482758343219757 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916142225265503 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16762453317642212 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07804655283689499 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9858964681žµóe625366 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0073964535258710384 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4431603252887726 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03084823675453663 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_6‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 <"Purity/*FRt1SRqqt2FBt/event_purity‚0.0073924739845097065 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1414007842540741 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38468995690345764 particle/accuracy_5_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18910618126392365 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34513887763023376 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4286481440067291 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 epoch‚3 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5466857552528381 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 _step‚395 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3187512755393982 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4387042224407196 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.16820989549160004 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1507202535867691 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5300594568252563 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07446808367967606 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4055176079273224 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2935185730457306 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.988315761089325 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5413914918899536 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.47410720586776733 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5364823341369629 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9911099076271057 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.49966931343078613 particle/accuracy_3_of_3‚1 ,particle/sensitivity‚0.24070312082767487 +jet/accuracy_2_of_6‚0.48076921701431274 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910277724266052 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21966707706451416 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21940483152866364 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2172521948814392 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887279868125916 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42945754528045654 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14863000810146332 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.21061860024929047 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2216050922870636 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.354166716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16883297264575958 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_2‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983001947402954 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.3241412341594696 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47667407989501953 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1114613264799118 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021328363567590714 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5069570541381836 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.221902534365654 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1960902065038681 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.013701708056032658 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4475308656692505 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42871004343032837 *jet/accuracy_1_of_1‚0.4382284879684448 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.34041476249694824 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13477367162704468 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978162050247192 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3599713146686554 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1914682388305664 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.45609140396118164 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3301742672920227 6 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.111111119389534 particle/accuracy_1_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4286932647228241 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35447436571121216 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916418194770813 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12997233867645264 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35459768772125244 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5086028575897217 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.41651278734207153 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883127808570862 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.33888891339302063 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1337965428829193 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006216931622475386 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32984232902526855 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4270089566707611 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2812349200248718 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15858443081378937 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4423665404319763 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19581663608551025 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3361598253250122 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6315467953681946 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4207279086112976 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5455032587051392 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.40948280692100525 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22614942491054535 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 jet/accuracy_6_of_7‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07499999552965164 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3389880657196045 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.007392473053187132 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.45823392271995544 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19895866513252258 *jet/accuracy_1_of_3‚0.8338066339492798 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1997302919626236 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855235815048218 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4647151231765747 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.395061731338501 jet/accuracy_6_of_6‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018083930015563965 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4100877344608307 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4148328900337219 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17280170321464539 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991620659828186 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913694262504578 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10572026669979095 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2198168784379959 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15116465091705322 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.39688941836357117 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2974388897418976 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9528481960296631 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07399965077638626 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3551217317581177 /"Purity/1FRt1SRqqt2FBt/event_purity‚0.015625 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_3_of_6‚0.09615384787321091 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1663619577884674 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1535172164440155 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1008652001619339 +jet/accuracy_3_of_4‚0.05758652091026306 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884328944327004e+09 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9710031151771545 particle/accuracy_1_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2609119415283203 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4148328900337219 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11854711174964905 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.995968759059906 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5131606459617615 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908711314201355 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32188552618026733 particle/accuracy_2_of_6‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020536573603749275 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015171571634709835 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4077672064304352 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961805582046509 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1805555671453476 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4610985219478607 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.954963743686676 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2567567527294159 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7142195105552673 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 )loss/val_total_loss‚46.19976912625947 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5803571343421936 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166865348816 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910231232643127 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.058532148599624634 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.23011694848537445 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35472118854522705 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15313144028186798 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028722601011395454 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7validation_average_jet_accuracy‚0.40109196305274963 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16781015694141388 5 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35305055975914 particle/accuracy_3_of_5‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 jet/accuracy_2_of_7‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05WàNÆÞ ¹¤ :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08517809212207794 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.20882493257522583 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04317142814397812 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8434349298477173 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.442631334066391 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 +jet/accuracy_2_of_2‚0.09015357494354248 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.423658162355423 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010552243329584599 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4748994708061218 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.003138292348012328 particle/accuracy_2_of_2‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17958088219165802 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.452562540769577 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5896391272544861 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01725367270410061 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.3227407932281494 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4300000071525574 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5147100687026978 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5961538553237915 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.30435436964035034 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12997233867645264 particle/accuracy_5_of_5‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21435414254665375 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33731013536453247 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31349167227745056 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 trainer/global_step‚4835 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15859845280647278 <"Purity/0FRt1SRqqt2FBt/event_purity‚0.0024630543775856495 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4262963831424713 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.048190534114837646 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2689529359340668 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915844798088074 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4379558265209198 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982104480266571 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4482758343219757 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916142225265503 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16762453317642212 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07804655283689499 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9858964681625366 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0073964535258710384 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4431603252887726 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03084823675453663 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_6‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 <"Purity/*FRt1SRqqt2FBt/event_purity‚0.0073924739845097065 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1414007842540741 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38468995690345764 particle/accuracy_5_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18910618126392365 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34513887763023376 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4286481440067291 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 epoch‚3 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5466857552528381 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 _step‚395 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3187512755393982 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4387042224407196 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_p¡ÙéXbwurity‚0.16820989549160004 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1507202535867691 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5300594568252563 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07446808367967606 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4055176079273224 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2935185730457306 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.988315761089325 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5413914918899536 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.47410720586776733 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5364823341369629 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9911099076271057 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.49966931343078613 particle/accuracy_3_of_3‚1 ,particle/sensitivity‚0.24070312082767487 +jet/accuracy_2_of_6‚0.48076921701431274 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910277724266052 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21966707706451416 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21940483152866364 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2172521948814392 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887279868125916 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42945754528045654 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14863000810146332 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.21061860024929047 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2216050922870636 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.354166716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16883297264575958 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_2‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983001947402954 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.3241412341594696 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47667407989501953 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1114613264799118 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021328363567590714 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5069570541381836 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.221902534365654 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1960902065038681 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.013701708056032658 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4475308656692505 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42871004343032837 *jet/accuracy_1_of_1‚0.4382284879684448 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.34041476249694824 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13477367162704468 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978162050247192 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3599713146686554 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1914682388305664 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.45609140396118164 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3301742672920227 6 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.111111119389534 particle/accuracy_1_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4286932647228241 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35447436571121216 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916418194770813 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12997233867645264 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35459768772125244 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5086028575897217 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.41651278734207153 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883127808570862 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.33888891339302063 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1337965428829193 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006216931622475386 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32984232902526855 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4270089566707611 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2812349200248718 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15858443081378937 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4423665404319763 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19581663608551025 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3361598253250122 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6315467953681946 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4207279086112976 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5455032587051392 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.40948280692100525 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22614942491054535 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 jet/accuracy_6_of_7‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07499999552965164 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3389880657196045 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.007392473053187132 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.45823392271995544 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19895866513252258 *jet/accuracy_1_of_3‚0.8338066339492798 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1997302919626236 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855235815048218 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4647151231765747 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.395061731338501 jet/accuracy_6_of_6‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018083930015563965 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4100877344608307 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4148328900337219 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17280170321464539 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991620659828186 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913694262504578 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10572026669979095 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2198168784379959 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15116465091705322 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.39688941836357117 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2974388897418976 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9528481960296631 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07399965077638626 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3551217317581177 /"Purity/1FRt1SRqqt2FBt/event_purity‚0.015625 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_3_of_6‚0.09615384787321091 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1663619577884674 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1535172164440155 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1008652001619339 +jet/accuracy_3_of_4‚0.05758652091026306 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884328944327004e+09 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9710031151771545 particle/accuracy_1_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2609119415283203 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4148328900337219 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11854711174964905 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.995968759059906 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5131606459617615 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908711314201355 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32188552618026733 particle/accuracy_2_of_6‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020536573603749275 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015171571634709835 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4077672064304352 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961805582046509 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1805555671453476 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4610985219478607 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.954963743686676 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2567567527294159 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7142195105552673 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 )loss/val_total_loss‚46.19976912625947 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5803571343421936 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166865348816 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910231232643127 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.058532148599624634 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.23011694848537445 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35472118854522705 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15313144028186798 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028722601011395454 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7validation_average_jet_accuracy‚0.40109196305274963 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16781015694141388 5 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35305055975914 particle/accuracy_3_of_5‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 jet/accuracy_2_of_7‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14704585075378418 particle/accuracy_1_of_3‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.40410569310188293 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16014830768108368 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4103584587574005 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02644987590610981 +jet/accuracy_3_of_5‚0.08962492644786835 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20274661481380463 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999701976776 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3916010856628418 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6684707403182983 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20912624895572662 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5625 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.33118146657943726 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14876677095890045 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010101309977471828 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17665420472621918 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20709428191184998 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.422171950340271 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956322312355042 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5321015119552612 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 _runtime‚4872.789952214 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988184571266174 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 particle/accuracy_7_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4317580759525299 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178707957267761 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4112611413002014 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5304415822029114 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.07575757801532745 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1997126340866089 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3373550474643707 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5337821841239929 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34259262681007385 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4599137604236603 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2613636255264282 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17637430131435394 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8920972943305969 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75‹¦‰Šº¡‚(:’ €¤åÔõ†”µ gpu.0.gpu‚14.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚36.0 gpu.0.powerWatts‚101.489' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.297800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÿ•^¢‚(:Ó €¤åÔŸ“î¶ disk.sda3.out‚ 183.109375 disk.sda1.in‚0 disk.rbd1.out‚2.453125) disk./.usagePercent‚18.70874387384211 disk.sda2.in‚0 disk.dm-0.in‚436.25 disk.drbd2080.out‚0 disk.dm-1.out‚ 183.109375$ memory_percent‚98.67262840270996$ disk./.usageGB‚278.7738037109375 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚35111.58935546875 disk.drbd1716.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 3968.7109375) proc.memory.percent‚24.22308921813965 network.sent‚ 1.1606429e+07 disk.sda.in‚0.203125 disk.sda.out‚ 183.109375 disk.drbd2265.out‚0' proc.memory.availableMB‚ 217.4765625 cpu‚25.287804163600548 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚436.25 disk.sda2.out‚0 disk.nvme0c0n1.in‚436.25 disk.drbd2080.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 network.recv‚ 1.751304e+06& disk.nvme0n1.out‚35111.58935546875 disk.drbd2265.in‚0# disk.dm-0.out‚35111.58935546875¤ªg2£‚(:’ ¤åÔøÊȵ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.025' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.80499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y@¼j㤂(:× ¤åÔ¨÷´·' disk.nvme0c0n1.out‚41009.1142578125 disk.sda.out‚ 183.97265625 disk.sda2.out‚0 disk.dm-1.out‚ 183.97265625" disk.dm-0.out‚41009.1142578125 disk.sda3.out‚ 183.97265625 disk.~GÇ.drbd1716.in‚0 disk.sda.in‚0.203125% disk.nvme0n1.out‚41009.1142578125) proc.memory.percent‚17.89226531982422 disk.nvme0c0n1.in‚436.25 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.5390625 disk.dm-0.in‚436.25 disk.nvme0n1.in‚436.25$ memory_percent‚72.71218299865723 cpu‚25.285902833216877 network.sent‚ 1.1729137e+07 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 network.recv‚ 1.757422e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 4470.8359375 proc.memory.rssMB‚ 2931.46875* disk./.usagePercent‚18.835331663290372$ disk./.usageGB‚280.6600532531738 disk.drbd2265.out‚0 proc.cpu.threads‚17 ËÂì2¥- _wandbruntime‚4898  _runtime‚4898ji£Ç¦Á % _timestamp‚1.7884329205110009e+09 _runtime‚4898.866068041 _step‚396 & lr-AdamW/pg1‚0.0001530949786663249 & lr-AdamW/pg2‚0.0001530949786663249 trainer/global_step‚4849ÐÏ©Ì§Æ & lr-AdamW/pg1‚0.0001530949786663249 & lr-AdamW/pg2‚0.0001530949786663249 trainer/global_step‚4849 % _timestamp‚1.7884329205110009e+09 _runtime‚4898.866068041 _step‚396Œ?Ž’2¨- _wandbruntime‚4898  _runtime‚4898û{$(©‚P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72663040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _step‚397 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104550075 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10355351575880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87162413 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44528640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚659375503371212 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87162407 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52166387 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14389969365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 4898.86692628 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52166390 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚564220940364288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660365131140096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38170572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚659378632268104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104573181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191712482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104573181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10369798384640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90336959 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578610982392832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10372903778304 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649023255264256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660365131140096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10352248106956 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38171048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670738061309952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191735594 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649023280692224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87162407 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670738061309952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660365157531648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚564220968498688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578610909729792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10369798384640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670734929524736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 $ _timestamp‚1.788432920511741e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87162413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90337438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10372903778304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14390013894144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878YÙi (ª‡P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 4898.86692628 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52166390 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚564220940364288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660365131140096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38170572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚659378632268104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104573181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191712482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104573181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10369798384640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90336959 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578610982392832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10372903778304 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649023255264256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660365131140096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10352248106956 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38171048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670738061309952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191735594 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649023280692224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87162407 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670738061309952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660365157531648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚564220968498688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578610909729792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10369798384640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670734929524736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 $ _timestamp‚1.788432920511741e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87162413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90337438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10372903778304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14390013894144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72663040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _step‚397 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104550075 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10355351575880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87162413 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44528640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚659375503371212 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87162407 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52166387 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14389969365504+CwÊ2«- _wandbruntime‚4899  _runtime‚4899耳»'¬ŽN T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670871667725312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649152677693440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87179691 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10354356900056 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660496651619840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191773994 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87179691 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14393021493248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191773994 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670868535926784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10375016105472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191750856 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74746880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659510138062628 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚4899.744783622 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660496651619840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90354768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649152652265472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87179697 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10371910698496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46612480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38178583 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 % _timestamp‚1.7884329213891144e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90355226 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52176643 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚564333351228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578726400856064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104594297 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚659507009165528 trainer/global_step‚4849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10371910698496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104594297 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚564333379362816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87179691 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10354356900056 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660496651619840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191773994 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87179691 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14393021493248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191773994 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670868535926784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10375016105472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191750856 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74746880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659510138062628 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚4899.744783622 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660496651619840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90354768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649152652265472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87179697 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10371910698496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46612480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38178583 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 % _timestamp‚1.7884329213891144e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90355226 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52176643 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚564333351228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578726400856064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104594297 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚659507009165528 trainer/global_step‚4849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10371910698496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104594297 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚564333379362816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 Bp²‚(:å ž¤åÔ¾ÿ´¸* disk./.usagePercent‚18.839862740638893( disk.nvme0c0n1.out‚42158.07568359375 disk.sda3.in‚0.203125& disk.nvme0n1.out‚42158.07568359375$ memory_percent‚72.71287441253662 disk.dm-0.in‚ 439.4765625 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 439.4765625 disk.dm-1.in‚0.203125$ disk./.usageGB‚280.7275695800781! disk.nvme0c0n1.in‚ 439.4765625 disk.sda3.out‚ 184.5078125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 cpu‚25.28512385675957 disk.rbd1.out‚ 2.5390625 disk.sda1.out‚0# disk.dm-0.out‚42158.07568359375 disk.sda.in‚0.203125 disk.dm-1.out‚ 184.5078125 network.sent‚ 1.1733431e+07 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2931.46875) proc.memory.percent‚17.89226531982422 disk.drbd1716.out‚0 disk.sda.out‚ 184.5078125 disk.rbd0.in‚0 disk.drbd2266.out‚0) proc.memory.availableMB‚ 4470.72265625 proc.cpu.threads‚17 network.recv‚ 1.76016e+06/nÎ$³‚(:’ ­¤åÔ뎂µ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À¶S©x´‚(:í ­¤åԟ伸# proc.memory.rssMB‚ 2931.58203125* proc.memory.percent‚17.892956733703613 cpu‚25.284410107693347 proc.cpu.threads‚17$ disk./.usageGB‚280.7776679992676& disk.nvme0n1.out‚42260.68896484375 disk.drbd2266.out‚0 disk.dm-0.in‚ 439.4765625 disk.sda3.in‚0.203125 disk.sda3.out‚ 185.12890625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.6015625 disk.drbd2265.in‚0 disk.rbd0.out‚0 network.sent‚ 1.1840248e+07 disk.nvme0n1.in‚ 439.4765625 disk.sda2.in‚0# disk.dm-0.out‚42260.68896484375 disk.sda.out‚ 185.12890625# memory_percent‚72.7146863937378 network.recv‚ 1.765667e+06* disk./.usagePercent‚18.843224887586068 disk.drbd2266.in‚0 disk.drbd2265.out‚0( disk.nvme0c0n1.out‚42260.68896484375 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 185.12890625! disk.nvme0c0n1.in‚ 439.4765625 disk.sda1.out‚0) proc.memory.availableMB‚ 4470.42578125Î)¯žµ‚(:“ ¼¤åÔ½Ýð´ gpu.0.gpu‚88.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚282.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.590399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Fx¶‚(:í ¼¤åÔ”Ѿ· disk.sda1.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 2.6015625 disk.dm-0.in‚ 439.4765625 disk.drbd1716.in‚0 disk.drbd2080.in‚0$ memory_percent‚72.71444797515869 network.sent‚ 1.1844865e+07 network.recv‚ 1.767993e+06! disk.nvme0c0n1.in‚ 439.4765625 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 185.63671875 cpu‚25.283711828634544 disk.sda3.in‚0.203125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚42329.34033203125 disk.sda1.in‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 4470.46484375) proc.memory.percent‚17.89419651031494* disk./.usagePercent‚18.847755708927053 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.sda3.out‚ 185.63671875 disk.nvme0n1.in‚ 439.4765625 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚42329.34033203125 disk.rbd0.in‚0& disk.nvme0n1.out‚42329.34033203125 disk.sda2.out‚0 disk.dm-1.out‚ 185.63671875# proc.memory.rssMB‚ 2931.78515625$ disk./.usageGB‚280.8451805114746îËÉr2·- _wandbruntime‚4945  _runtime‚4945åT8ɸà ' lr-AdamW/pg2‚0.00015180819048800942 trainer/global_step‚4899 % _timestamp‚1.7884329671561239e+09 _runtime‚4945.511202473 _step‚400 ' lr-AdamW/pg1‚0.00015180819048800942ËØ Î¹È trainer/global_step‚4899 % _timestamp‚1.7884329671561239e+09 _runtime‚4945.511202473 _step‚400 ' lr-AdamW/pg1‚0.00015180819048800942 ' lr-AdamW/pg2‚0.00015180819048800942l?… 2º- _wandbruntime‚4945  _runtime‚4945¿Š(»‚P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666939835549696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91235454 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38555894 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193634534 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666939861941248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚665950806678220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10475428491264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚677415264040960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105607927 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4899 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚655493105325056 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚677418395826176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10460804822088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14538546484224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10475428491264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105631033 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52679040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10478533884928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88026607 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105631033 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚677418395826176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _step‚401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88026613 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52679037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚584380111106560 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105607927 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 # _timestamp‚1.78843296715695e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88026607 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14538599401472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚665953935575112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193657646 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚4945.512238838 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚677415264040960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91234931 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚569841511705088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚569841483570688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10457701353164 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666939861941248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666939835549696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10478533884928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666939861941248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚665950806678220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10475428491264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚677415264040960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105607927 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4899 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freeà”‚”Æd‚655493105325056 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚677418395826176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10460804822088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14538546484224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10475428491264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105631033 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52679040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10478533884928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88026607 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105631033 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚677418395826176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _step‚401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88026613 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52679037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚584380111106560 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105607927 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 # _timestamp‚1.78843296715695e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88026607 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14538599401472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚665953935575112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193657646 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚4945.512238838 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚677415264040960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91234931 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚569841511705088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚569841483570688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10457701353164‘çâO 2½- _wandbruntime‚4946  _runtime‚4946Ùù'¾ŽN YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10459810146264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105652149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667071356029440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚677552002241536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105629017 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14541496150016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88043897 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10480646212096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193696046 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91252688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52689293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38563882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38563398 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667071356029440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91253175 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193672908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚677552002241536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666085441369636 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666082312472536 trainer/global_step‚4899 _runtime‚4946.398623729 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193672908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚569953894434816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655622527754240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667071329637888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10462913615396 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193696046 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚584495390584832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105629017 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667071329637888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584495471623168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10477540805120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920{iŠù'¿“N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10480646212096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193696046 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91252688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52689293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38563882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38563398 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667071356029440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91253175 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193672908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚677552002241536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666085441369636 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666082312472536 trainer/global_step‚4899 _runtime‚4946.398623729 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193672908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚569953894434816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655622527754240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667071329637888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10462913615396 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193696046 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚584495390584832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105629017 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667071329637888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584495471623168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10477540805120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10459810146264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105652149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667071356029440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚677552002241536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105629017 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14541496150016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88043897 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440’”Þ#2À- _wandbruntime‚4946  _runtime‚4946(Ô›XÁÒ /loss/FRt4/detection_loss‚3.1979813814163207 .loss/FBt2/detection_loss‚2.368401919092451 2loss/SRqqt2/assignment_loss‚1.9002878139061588 2loss/SRqqt3/assignment_loss‚1.3392926069022513 2loss/SRqqt4/assignment_loss‚1.3694104534489187 .loss/FBt3/detection_loss‚3.410394157682146 % _timestamp‚1.7884329680450463e+09 0loss/FRt3/assignment_loss‚2.3798627780408275 0loss/FBt2/assignment_loss‚0.3059220654214414 0loss/FBt4/assignment_loss‚0.3095071158447051 _runtime‚4946.400030793 0loss/SRqqt2/detection_loss‚2.476720655603068 epoch‚4 2loss/SRqqt1/assignment_loss‚1.3336385352339484 .loss/FRt2/detection_loss‚2.351416440237136 1loss/SRqqt1/detection_loss‚1.1422178549409816 1loss/SRqqt4/detection_loss‚2.6268891245126724 /loss/FBt1/detection_loss‚0.9060853724159411 /loss/FRt1/assignment_loss‚1.924148824933457 /loss/FRt4/assignment_loss‚4.061658754348755 1loss/FBt1/assignment_loss‚0.08320143147469765 /loss/FRt3/detection_loss‚1.9758481241914692 0loss/SRqqt3/detection_loss‚1.238500927609426 /loss/FBt4/detection_loss‚1.8758856356143951 /loss/FRt2/assignment_loss‚4.339916938827152 _step‚403 -loss/FRt1/detection_loss‚1.51011371990181 trainer/global_step‚4899 %loss/total_loss‚44.43935066177247 2loss/FBt3/assignment_loss‚0.012048030172341637þRž]Â× _step‚403 -loss/FRt1/detection_loss‚1.51011371990181 trainer/global_step‚4899 %loss/total_loss‚44.43935066177247 2loss/FBt3/assignment_loss‚0.012048030172341637 /loss/FRt4/detection_loss‚3.1979813814163207 .loss/FBt2/detection_loss‚2.368401919092451 2loss/SRqqt2/assignment_loss‚1.9002878139061588 2loss/SRqqt3/assignment_loss‚1.3392926069022513 2loss/SRqqt4/assignment_loss‚1.3694104534489187 .loss/FBt3/detection_loss‚3.410394157682146 % _timestamp‚1.7884329680450463e+09 0loss/FRt3/assignment_loss‚2.3798627780408275 0loss/FBt2/assignment_loss‚0.3059220654214414 0loss/FBt4/assignment_loss‚0.3095071158447051 _runtime‚4946.400030793 0loss/SRqqt2/detection_loss‚2.476720655603068 epoch‚4 2loss/SRqqt1/assignment_loss‚1.3336385352339484 .loss/FRt2/detection_loss‚2.351416440237136 1loss/SRqqt1/detection_loss‚1.1422178549409816 1loss/SRqqt4/detection_loss‚2.6268891245126724 /loss/FBt1/detection_loss‚0.9060853724159411 /loss/FRt1/assignment_loss‚1.924148824933457 /loss/FRt4/assignment_loss‚4.061658754348755 1loss/FBt1/assignment_loss‚0.08320143147469765 /loss/FRt3/detection_loss‚1.9758481241914692 0loss/SRqqt3/detection_loss‚1.238500927609426 /loss/FBt4/detection_loss‚1.8758856356143951 /loss/FRt2/assignment_loss‚4.339916938827152“Õ3’Â(:’ ˤåÔÞ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚284.309' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.86180000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZlämyÄ‚(:î ˤåÔËÛî·$ disk./.usageGB‚280.8980941772461 disk.rbd1.out‚ 2.6337890625! disk.nvme0c0n1.in‚ 439.4765625 cpu‚25.282952996184868 proc.cpu.threads‚17 disk.rbd1.in‚0( disk.nvme0c0n1.out‚42441.85595703125 disk.dm-0.in‚ 439.4765625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.sent‚ 1.1850083e+07 disk.nvme0n1.in‚ 439.4765625 disk.sda2.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0$ memory_percent‚72.71497249603271# disk.dm-0.out‚42441.85595703125" proc.memory.rssMB‚ 2931.8359375 network.recv‚ 1.771921e+06 disk.drbd2266.out‚0 disk.sda3.out‚ 186.33203125 disk.drbd1716.in‚0 disk.sda.out‚ 186.33203125 disk.drbd2080.in‚0 disk.drbd2266.in‚0( disk./.usagePercent‚18.8513067894345 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚42441.85595703125 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 186.33203125 disk.drbd2080.out‚0) proc.memory.availableMB‚ 4470.37890625* proc.memory.percent‚17.894506454467773žÅ‚(:“ Ú¤åÔŸÔþ´ gpu.0.gpu‚92.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚298.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pïÜÌwÆ‚(:ì Ú¤åÔí¯á· disk.rbd1.out‚ 2.6337890625 cpu‚25.28225389018287 proc.cpu.threads‚17 disk.sda3.in‚0.203125 disk.sda3.out‚ 186.8828125 disk.sda.out‚ 186.8828125 disk.drbd2265.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚42504.50927734375 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 439.4765625 disk.dm-0.in‚ 439.4765625 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 439.4765625& disk.nvme0n1.out‚42504.50927734375 disk.dm-1.out‚ 186.8828125 disk.rbd1.in‚0$ memory_percent‚72.71676063537598 network.sent‚ 1.1955825e+07$ disk./.usageGB‚280.9569511413574 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0( proc.memory.availableMB‚ 4470.0859375* disk./.usagePercent‚18.855256729680292 disk.drbd1716.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2932.12890625* proc.memory.percent‚17.896294593811035 network.recv‚ 1.776666e+06# disk.dm-0.out‚42504.50927734375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0!ÚÌޞǂ(:“ é¤åԈؽµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.448' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.889599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cu¾FaÈ‚(:Ö é¤åÔ¦¾À· disk.sda3.in‚0.203125 disk.sda2.in‚0 disk.dm-1.out‚187.375 disk.drbd2266.out‚0$ memory_percent‚72.71995544433594 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 439.4765625 cpu‚25.281560574892367 disk.sda.out‚187.375 disk.sda3.out‚187.375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.dm-0.out‚42627.48828125% proc.memory.availableMB‚ 4469.5625* disk./.usagePercent‚18.858831106873357$ disk./.usageGB‚281.0102119445801 disk.nvme0n1.in‚ 439.4765625# disk.nvme0n1.out‚42627.48828125 disk.rbd1.out‚ 2.69677734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1962498e+07 network.recv‚ 1.781499e+06 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0% disk.nvme0c0n1.out‚42627.48828125 disk.dm-0.in‚ 439.4765625# proc.memory.rssMB‚ 2932.40234375 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.897963523864746Ç’žY2É- _wandbruntime‚4992  _runtime‚4992Ó=¡Þ/Êà _runtime‚4992.283954113 _step‚Λq>š404 ' lr-AdamW/pg1‚0.00015051108179547503 ' lr-AdamW/pg2‚0.00015051108179547503 trainer/global_step‚4949 % _timestamp‚1.7884330139288888e+09ã¹>IÎËÈ % _timestamp‚1.7884330139288888e+09 _runtime‚4992.283954113 _step‚404 ' lr-AdamW/pg1‚0.00015051108179547503 ' lr-AdamW/pg2‚0.00015051108179547503 trainer/global_step‚4949”[ 5†2Ì- _wandbruntime‚4992  _runtime‚4992÷óHQ (Í„P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14686742258688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884330139296348e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88890807 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14686691438592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4992.284885891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673514566350848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚661962955385856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10584163991552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590148718215680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38939631 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106665779 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53191690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10581058597888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673514539959296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10584163991552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590148797170176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38939110 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684095598557184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195579698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚684098730342400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88890807 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53191687 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚405 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92130797 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88890813 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575462026777088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚661962980813824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672526109985228 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚4949 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684098730342400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684095598557184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88890813 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180©ÕTµ(ΉP VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684098730342400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684095598557184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88890813 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14686742258688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884330139296348e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88890807 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14686691438592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4992.284885891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673514566350848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚661962955385856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10584163991552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590148718215680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38939631 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106665779 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53191690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10581058597888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673514539959296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10584163991552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590148797170176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38939110 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684095598557184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195579698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚684098730342400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88890807 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53191687 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚405 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92130797 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88890813 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575462026777088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚661962980813824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672526109985228 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚4949•ÿwÌÞ2Ï- _wandbruntime‚4993  _runtime‚4993tQã'ÐN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚4949 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106686869 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4993.168588443 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10586276318720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10583170911744 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92149203 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚590264096311808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14689658670592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662092352387072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92148706 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195594960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10565263392472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88908097 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788433014812722e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195618098 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106710001 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10586276318720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672657615779544 _step‚406 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88908091 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10583170911744 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106686869 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662092377815040 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10568366861604 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88908097 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673646060439040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672660744676644 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673646034047488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575574465775616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575574437641216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684232336757760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚673646034047488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53201940 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14689709477376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684232336757760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195618098 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53201943 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚673646060439040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88908091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38947260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195594960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590264175252992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684229204959232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚497 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788433014812722e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195618098 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106710001 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10586276318720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672657615779544 _step‚406 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88908091 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10583170911744 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106686869 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662092377815040 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10568366861604 M>DeviceStþyðH atsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88908097 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673646060439040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672660744676644 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673646034047488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575574465775616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575574437641216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684232336757760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚673646034047488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53201940 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14689709477376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684232336757760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195618098 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53201943 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚673646060439040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88908091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38947260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195594960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590264175252992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684229204959232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚497 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4993.168588443 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10586276318720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10583170911744 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92149203 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚590264096311808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14689658670592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662092352387072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92148706 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195594960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10565263392472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88908097 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690775933073408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10686688704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680089244368896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690779064858624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690779064858624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10689794098176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚680089244368896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53704340 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14834565934592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10689794098176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10671711314504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚5038.977544045 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107746737 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10686688704512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚581082569983488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚679101413292236 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690775933073408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚668432830874624 trainer/global_step‚4999 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89755007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 % _timestamp‚1.7884330606222599e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878,9©2à‰P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690779064858624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10689794098176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚680089244368896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53704340 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14834565934592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10689794098176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10671711314504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚5038.977544045 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107746737 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10686688704512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚581082569983488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚679101413292236 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690775933073408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚668432830874624 trainer/global_step‚4999 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89755007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 % _timestamp‚1.7884330606222599e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93027745 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690775933073408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10686688704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680089244368896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690779064858624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312™í{2á- _wandbruntime‚5039  _runtime‚5039éŒÿ 'âŽN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53714590 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚596032448330240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197517012 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197540150 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚581195008982016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39330286 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89772291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14837467482624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10673820107812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10691906425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93045388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53714593 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _runtime‚5039.862066184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197540150 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89772297 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10691906425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39330795 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10688801018368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚679236047983652 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680220738457088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884330615066025e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107767853 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14837520386560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚679232919086552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107744721 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680220738457088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690912671273984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107744721 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89772297 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690909539475456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10688801018368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93044876 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚596032529368576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚581194980847616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89772291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14837467482624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10673820107812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10691906425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93045388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53714593 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _runtime‚5039.862066184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197540150 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89772297 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10691906425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39330795 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10688801018368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚679236047983652 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680220738457088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884330615066025e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107767853 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14837520386560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚679232919086552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107744721 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680220738457088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690912671273984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107744721 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89772297 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690909539475456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10688801018368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93044876 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚596032529368576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚581194980847616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚581195008982016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39330286 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878šqŽÐÑ2ä- _wandbruntime‚5039  _runtime‚5039>kâ˜YåÓ 2loss/SRqqt2/assignment_loss‚2.0755904841990698 1loss/SRqqt3/detection_loss‚1.5000299760563807 0loss/FRt4/assignment_loss‚4.1774863569360035 .loss/FRt3/detection_loss‚2.065612868490544 .loss/FRt4/detection_loss‚3.660696870402286 0loss/SRqqt1/detection_loss‚1.345098114713575 /loss/FBt3/detection_loss‚1.9474318098080785 .loss/FBt4/detection_loss‚1.660659474907098 trainer/global_step‚4999 _runtime‚5039.863117367 /loss/FRt2/assignment_loss‚3.671044518870692 0loss/FBt4/assignment_loss‚0.2523171119542564 0loss/FRt1/assignment_loss‚2.3918897922961944 2loss/SRqqt1/assignment_loss‚1.3305378136150963 2loss/FBt1/assignment_loss‚0.061686504038265706 1loss/FBt3/assignment_loss‚0.14323453759463314 /loss/FRt1/detection_loss‚1.7382322468103901 1loss/SRqqt4/detection_loss‚2.8561967611312866 epoch‚4 1loss/SRqqt3/assignment_loss‚1.307858762599415 1loss/FBt2/assignment_loss‚0.39148392325237663 .loss/FRt2/detection_loss‚3.040652867286436 %loss/total_loss‚45.75883171868181 _step‚411 0loss/FRt3/assignment_loss‚2.2901407290588724 0loss/SRqqt2/detection_loss‚2.649571969395592 .loss/FBt2/detection_loss‚2.625584000035336 % _timestamp‚1.7884330615081975e+09 2loss/SRqqt4/assignment_loss‚1.4573700546341783 .loss/FBt1/detection_loss‚1.118424170595758Fâ½^æØ 1loss/SRqqt3/assignment_loss‚1.307858762599415 1loss/FBt2/assignment_loss‚0.39148392325237663 .loss/FRt2/detection_loss‚3.040652867286436 %loss/total_loss‚45.75883171868181 _step‚411 0loss/FRt3/assignment_loss‚2.2901407290588724 0loss/SRqqt2/detection_loss‚2.649571969395592 .loss/FBt2/detection_loss‚2.625584000035336 % _timestamp‚1.7884330615081975e+09 2loss/SRqqt4/assignment_loss‚1.4573700546341783 .loss/FBt1/detection_loss‚1.118424170595758 2loss/SRqqt2/assignment_loss‚2.0755904841990698 1loss/SRqqt3/detection_loss‚1.5000299760563807 0loss/FRt4/assignment_loss‚4.1774863569360035 .loss/FRt3/detection_loss‚2.065612868490544 .loss/FRt4/detection_loss‚3.660696870402286 0loss/SRqqt1/detection_loss‚1.345098114713575 /loss/FBt3/detection_loss‚1.9474318098080785 .loss/FBt4/detection_loss‚1.660659474907098 trainer/global_step‚4999 _runtime‚5039.863117367 /loss/FRt2/assignment_loss‚3.671044518870692 0loss/FBt4/assignment_loss‚0.2523171119542564 0loss/FRt1/assignment_loss‚2.3918897922961944 2loss/SRqqt1/assignment_loss‚1.3305378136150963 2loss/FBt1/assignment_loss‚0.061686504038265706 1loss/FBt3/assignment_loss‚0.14323453759463314 /loss/FRt1/detection_loss‚1.7382322468103901 1loss/SRqqt4/detection_loss‚2.8561967611312866 epoch‚4›ëóäÆ“ç‚(:ˆ ¥¥åÔ¡²Ïµ gpu.0.gpu‚85.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚127.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§.‹iè‚(:Þ ¥¥åÔãã©·) disk./.usagePercent‚18.87104471431689 disk.nvme0n1.in‚ 439.4765625 disk.drbd2266.in‚0 disk.sda3.out‚ 189.58203125 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 189.58203125 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.drbd2266.out‚0$ disk.nvme0c0n1.out‚ 42919.3203125 disk.drbd1716.in‚0" memory_percent‚72.725510597229$ disk./.usageGB‚281.1922035217285 disk.rbd1.out‚ 2.759765625! disk.nvme0c0n1.in‚ 439.4765625 disk.drbd2080.out‚0 network.sent‚ 1.2083976e+07 disk.sda2.in‚0 disk.dm-0.in‚ 439.4765625 disk.drbd1716.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.90318489074707 cpu‚25.27877039816589 network.recv‚ 1.796928e+06 disk.rbd1.in‚0 disk.dm-0.out‚ 42919.3203125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.in‚0" disk.nvme0n1.out‚ 42919.3203125 disk.sda1.out‚0 disk.sda.out‚ 189.58203125 disk.drbd2265.in‚0) proc.memory.availableMB‚ 4468.65234375" proc.memory.rssMB‚ 2933.2578125­‘Éí“é‚(:ˆ ´¥åÔÕùŵ gpu.0.gpu‚81.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚302.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚029_oê‚(:ä ´¥åÔ†„ê· disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 cpu‚25.278040586721755 disk.sda3.in‚0.203125 disk.sda3.out‚ 189.9453125 disk.drbd2265.out‚0$ memory_percent‚72.72791862487793* disk./.usagePercent‚18.874512848383247% disk.nvme0c0n1.out‚42999.62890625 disk.nvme0n1.in‚ 439.4765625% disk./.usageGB‚281.24388122558594 disk.sda.out‚ 189.9453125 disk.drbd2265.in‚0( proc.memory.availableMB‚ 4468.2578125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.905306816101074 proc.cpu.threads‚17 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2933.60546875 network.recv‚ 1.801099e+06 disk.dm-0.in‚ 439.4765625 disk.dm-0.out‚42999.62890625 disk.sda1.in‚0 network.sent‚ 1.2189305e+07 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625# disk.nvme0n1.out‚42999.62890625 disk.sda2.in‚0 disk.dm-1.out‚ 189.9453125! disk.nvme0c0n1.in‚ 439.4765625!d“ë‚(:ˆ Ã¥åÔ«Í…¶ gpu.0.gpu‚50.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ý0#^ì‚(:Ó Ã¥åÔɤ°· disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 190.59375 disk.drbd1716.in‚0) proc.memory.percent‚17.90645122528076 disk.sda.out‚ 190.59375 disk.rbd0.out‚0 disk.dm-0.out‚ 43064.09375# memory_percent‚72.7276086807251# proc.memory.rssMB‚ 2933.79296875 network.sent‚ 1.2195488e+07 disk.sda2.out‚0 disk.dm-0.in‚ 439.4765625 cpu‚25.277373990814635 proc.cpu.threads‚17* disk./.usagePercent‚18.877846066479055 disk.sda.in‚0.203125 disk.nvme0n1.out‚ 43064.09375 disk.dm-1.out‚ 190.59375 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.out‚ 43064.09375) proc.memory.availableMB‚ 4468.30859375 disk.sda2.in‚0 network.recv‚ 1.805765e+06 disk.drbd2265.out‚0 disk.drbd2266.in‚0! disk.nvme0c0n1.in‚ 439.4765625$ disk./.usageGB‚281.2935485839844 disk.nvme0n1.in‚ 439.4765625ùés“í‚(:ˆ Ò¥åÔˆµ¹µ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚258.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àðî‚(:é Ò¥åÔìÛš·$ disk./.usageGB‚281.3428268432617 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚43163.30517578125 disk.drbd2265.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚43163.30517578125 cpu‚25.276757417253314 network.recv‚ 1.809837e+06 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 439.4765625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 439.4765625 proc.cpu.threads‚17) disk./.usagePercent‚18.88115317180637 disk.drbd2265.out‚0! disk.nvme0c0n1.in‚ 439.4765625# disk.dm-0.out‚43163.30517578125' proc.memory.availableMB‚ 4468.453125 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.203125 disk.rbd’¼ZÀ„0.in‚0 disk.dm-1.out‚ 190.9296875$ memory_percent‚72.72672653198242 network.sent‚ 1.2201308e+07 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.sda.out‚ 190.9296875 disk.rbd1.in‚0 disk.sda3.out‚ 190.9296875 disk.sda1.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.907094955444336 disk.sda2.out‚0" proc.memory.rssMB‚ 2933.8984375…·6]2ï- _wandbruntime‚5085  _runtime‚5085Mó^ÔÇðÁ trainer/global_step‚5049 % _timestamp‚1.7884331070562687e+09 _runtime‚5085.411312464 _step‚412 & lr-AdamW/pg1‚0.0001478869862242254 & lr-AdamW/pg2‚0.0001478869862242254#-qÌñÆ & lr-AdamW/pg1‚0.0001478869862242254 & lr-AdamW/pg2‚0.0001478869862242254 trainer/global_step‚5049 % _timestamp‚1.7884331070562687e+09 _runtime‚5085.411312464 _step‚412œzŸÅ»2ò- _wandbruntime‚5085  _runtime‚5085«t;Ñ (ó„P eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586703141324288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚697456267589632 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10774061091788 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686663948778496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10792318811136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14982478590464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884331070570066e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586703113189888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚674902680935424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39704996 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90619213 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10795424204800 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚697459399374848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90619207 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685679845496136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10777164560712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601685619914752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54216990ß¾C(ô‰P dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14982478590464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884331070570066e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586703113189888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚674902680935424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39704996 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90619213 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10795424204800 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚697459399374848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90619207 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685679845496136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10777164560712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601685619914752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54216990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586703141324288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚697456267589632 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10774061091788 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686663948778496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10792318811136ñB—2õ- _wandbruntime‚5086  _runtime‚5086'çñ¿'öŽN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686795442866688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90636491 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586815524054016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108825705 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90636497 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚697593005790208 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚686795442866688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14985385236480 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199462202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚500 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586815552188416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10797536531968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54227240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601800909290496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685808222393560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108802573 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199439064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686795469258240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108825705 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _runtime‚5086.296424263 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _step‚414 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10776169884888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90636497 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675032077936640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10797536531968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54227243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93939461 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108802573 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90636491 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10794431124992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14985438140416 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440Ó±}à÷“N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686795442866688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90636491 ûx™†9"I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586815524054016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108825705 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90636497 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚697593005790208 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚686795442866688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14985385236480 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199462202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚500 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586815552188416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10797536531968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54227240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601800909290496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685808222393560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108802573 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199439064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686795469258240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108825705 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _runtime‚5086.296424263 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _step‚414 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10776169884888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90636497 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675032077936640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10797536531968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54227243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93939461 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108802573 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90636491 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10794431124992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14985438140416 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704136602105856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693238653188096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681372530996224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94817665 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 % _timestamp‚1.7884331540632977e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚693238679579648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15130233217536 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109862441 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704136602105856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681372505568256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54729637 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692255148803144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10897948917760 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚417 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10879514337996 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15130182397440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704139733891072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10901054311424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91483413 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109862441 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109839335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚592323684530688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40088028 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91483413 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10901054311424 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201345854 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94818169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚5132.418556796 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40088529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10882617806920ê¾$†‰P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201345854 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94818169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚5132.418556796 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40088529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10882617806920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692252019906252 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607453917748224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚592323656396288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚607453838793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704139733891072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704136602105856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693238653188096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681372530996224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94817665 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 % _timestamp‚1.7884331540632977e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚693238679579648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_?–¸ñëbytes.small_pool.allocated‚15130233217536 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109862441 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704136602105856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681372505568256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54729637 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692255148803144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10897948917760 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚417 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10879514337996 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15130182397440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704139733891072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10901054311424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91483413 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109862441 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109839335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚592323684530688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40088028 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91483413 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10901054311424 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664¡‚—°â2‡- _wandbruntime‚5133  _runtime‚5133ûï'ˆŽN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15133201005568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _step‚418 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5099 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884331549522436e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚592436095394816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10900061231616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94836049 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681501902569472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10881623131096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91500697 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10884726600228 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201384254 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681501927997440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109883557 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201361116 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693370173667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91500691 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15133150198784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚693370173667840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40096156 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10903166638592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607569217459200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201361116 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91500691 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91500697 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94835572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚693370147276288 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10903166638592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109860425 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201384254 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚693370147276288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10900061231616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109883557 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54739893 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94835572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚693370147276288 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10903166638592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109860425 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201384254 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚693370147276288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10900061231616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109883557 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54739893 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _step‚418 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5099 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884331549522436e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚592436095394816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10900061231616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94836049 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681501902569472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10881623131096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91500697 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10884726600228 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201384254 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681501927997440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109883557 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201361116 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693370173667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91500691 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15133150198784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚693370173667840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40096156 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10903166638592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607569217459200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201361116 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91500691 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91500697 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10984967584204 % _timestamp‚1.7884332008390675e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55242290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10988071053128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11003579024384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92347607 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710816936622080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40470186*(˜‰P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55242287 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95712473 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15278064282624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11006684418048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597944227737088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597944199602688 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92347607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110897187 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11003579024384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚687842355629056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚5179.194264251 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11006684418048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚699813357597696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10984967584204 % _timestamp‚1.7884332008390675e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55242290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10988071053128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11003579024384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92347607 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710816936622080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40470186 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710820068407296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699813383989248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698830452110152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203244794 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698827323213260 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699813383989248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95713026 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613222344936960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613222263885312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92347613 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15278117199872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40470736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚421 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203267906 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚699813357597696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92347613¥'†bÅ2™- _wandbruntime‚5180  _runtime‚5180ˆ§¿µ'šŽN ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11008796745216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884332017219882e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203283168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710953674822656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110918277 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚687971752630272 _runtime‚5180.077614786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203306306 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613337640991232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10990179846436 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15281030524416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55252540 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15281083428352 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11008796745216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110941409 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92364891 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40477929 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598056610466816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚698958829007576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95730469 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚687971778058240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10987076377304 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203283168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110918277 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚5149 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11005691338240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55252543 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95730994 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚699944851685888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92364897 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚699944878077440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11005691338240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710950543024128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92364897 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613337722029568 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203306306 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180TþÖ(¸›“N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚5149 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11005691338240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55252543 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95730994 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚699944851685888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92364897 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚699944878077440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11005691338240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710950543024128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92364897 PADeviceStatsMonitor.on_traJÑiain_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613337722029568 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203306306 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11008796745216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884332017219882e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203283168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710953674822656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110918277 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚687971752630272 _runtime‚5180.077614786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203306306 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613337640991232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10990179846436 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15281030524416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55252540 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15281083428352 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11008796745216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110941409 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92364891 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40477929 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598056610466816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚698958829007576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95730469 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚687971778058240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10987076377304 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203283168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110918277 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440¦»É2œ- _wandbruntime‚5180  _runtime‚5180Ÿ„ÿ[Õ 0loss/SRqqt4/detection_loss‚2.435343250632286 -loss/FBt3/detection_loss‚2.30613317489624 /loss/FRt2/detection_loss‚2.4076137380166487 /loss/FBt2/detection_loss‚3.1231223995035347 0loss/FRt1/assignment_loss‚1.7646980437021407 2loss/SRqqt2/assignment_loss‚1.8669658288773563 /loss/FRt1/detection_loss‚1.8598279034097989 0loss/SRqqt1/detection_loss‚1.165649311397882 epoch‚4 0loss/FRt4/assignment_loss‚3.4827196037068084 2loss/SRqqt1/assignment_loss‚1.2159368027111186 1loss/FBt2/assignment_loss‚0.18077553866557786 1loss/FBt3/assignment_loss‚0.23271177992901357 1loss/FBt4/assignment_loss‚0.20796605889918282 0loss/SRqqt2/detection_loss‚2.708679488411656 trainer/global_step‚5149 % _timestamp‚1.7884332017237816e+09 0loss/FRt3/assignment_loss‚2.7350371281305947 2loss/SRqqt3/assignment_loss‚1.1883699318796799 /loss/FRt3/detection_loss‚2.1856810832741083 1loss/SRqqt3/detection_loss‚1.4718303505230594 %loss/total_loss‚45.74950710651528 _runtime‚5180.078747623 2loss/SRqqt4/assignment_loss‚2.0006156265735626 /loss/FBt1/detection_loss‚0.9218388820959565 _step‚423 .loss/FBt4/detection_loss‚2.381141263246536 /loss/FRt2/assignment_loss‚4.095639982006767 1loss/FBt1/assignment_loss‚0.20195892069921736 .loss/FRt4/detection_loss‚3.609251015326556b$)`žÚ 0loss/FRt3/assignment_loss‚2.7350371281305947 2loss/SRqqt3/assignment_loss‚1.1883699318796799 /loss/FRt3/detection_loss‚2.1856810832741083 1loss/SRqqt3/detection_loss‚1.4718303505230594 %loss/total_loss‚45.74950710651528 _runtime‚5180.078747623 2loss/SRqqt4/assignment_loss‚2.0006156265735626 /loss/FBt1/detection_loss‚0.9218388820959565 _step‚423 .loss/FBt4/detection_loss‚2.381141263246536 /loss/FRt2/assignment_loss‚4.095639982006767 1loss/FBt1/assignment_loss‚0.20195892069921736 .loss/FRt4/detection_loss‚3.609251015326556 0loss/SRqqt4/detection_loss‚2.435343250632286 -loss/FBt3/detection_loss‚2.30613317489624 /loss/FRt2/detection_loss‚2.4076137380166487 /loss/FBt2/detection_loss‚3.1231223995035347 0loss/FRt1/assignment_loss‚1.7646980437021407 2loss/SRqqt2/assignment_loss‚1.8669658288773563 /loss/FRt1/detection_loss‚1.8598279034097989 0loss/SRqqt1/detection_loss‚1.165649311397882 epoch‚4 0loss/FRt4/assignment_loss‚3.4827196037068084 2loss/SRqqt1/assignment_loss‚1.2159368027111186 1loss/FBt2/assignment_loss‚0.18077553866557786 1loss/FBt3/assignment_loss‚0.23271177992901357 1loss/FBt4/assignment_loss‚0.20796605889918282 0loss/SRqqt2/detection_loss‚2.708679488411656 trainer/global_step‚5149 % _timestamp‚1.7884332017237816e+09§påö©Ÿ‚(:’ »¦åÔ¾¬”µ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚258.599' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.71979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aýyBe ‚(:Ú »¦åÔðÍŠ¸ disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1938.61328125) proc.memory.percent‚17.91064739227295 network.recv‚ 1.835717e+06% disk./.usageGB‚279.98071670532227 disk.sda.in‚0.203125 proc.cpu.threads‚17 network.sent‚ 1.2438787e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 443.609375 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 443.609375$ disk.nvme0n1.out‚43778.771484375 disk.nvme0c0n1.in‚ 443.609375 disk.sda1.in‚0 disk.drbd2080.out‚0 cpu‚25.272039817497603 disk.rbd0.in‚0& disk.nvme0c0n1.out‚43778.771484375 disk.sda3.in‚0.203125 disk.sda3.out‚196.1875 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 2.759765625 disk.drbd1716.in‚0 disk.drbd2080.in‚0* disk./.usagePercent‚18.789740817562723 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚196.1875! disk.dm-0.out‚43778.771484375# memory_percent‚88.1676435470581 disk.sda.out‚196.1875 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2934.48046875a=œ"¡‚(:’ ʦåÔ¤×âµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚281.595' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.31900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïIâs¢‚(:è ʦåÔð‡ž¸ disk.dm-0.out‚43880.68359375% disk./.usageGB‚280.06919860839844 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 465.80078125% disk.nvme0c0n1.out‚43880.68359375 disk.sda3.in‚0.203125# proc.memory.rssMB‚ 2934.66015625 network.sent‚ 1.2545704e+07 disk.sda.in‚0.203125 disk.rbd1.out‚ 2.759765625 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 465.80078125 disk.rbd1.in‚0* proc.memory.percent‚17.911744117736816 cpu‚25.271353616656754 network.recv‚ 1.841111e+06 disk.sda3.out‚ 197.12890625 disk.drbd1716.in‚0' proc.memory.availableMB‚ 1939.078125 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0# memory_percent‚88.1648063659668 disk.dm-0.in‚ 465.80078125 disk.sda.out‚ 197.12890625 disk.drbd2265.out‚0# disk.nvme0n1.out‚43880.68359375 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 197.12890625 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.795678912319428 disk.drbd2080.in‚0¼³|Ä£‚(:’ Ù¦åÔìäç´ gpu.0.gpu‚73.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚275.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.17759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â ÿ·q¤‚(:æ Ù¦åÔ°Ñ…¸ disk.drbd2266.out‚0% disk.nvme0c0n1.out‚43983.63671875 network.recv‚ 1.84374e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 465.83203125# disk.nvme0n1.out‚43983.63671875 disk.sda.in‚0.203125% disk./.usageGB‚280.11204528808594 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.759765625& proc.memory.availableMB‚ 1938.84375 disk.rbd1.in‚0 disk.sda3.in‚0.203125# memory_percent‚88.1662368774414 disk.drbd2080.in‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2934.66015625 disk.drbd2266.in‚0 disk.sda1.in‚0 network.sent‚ 1.2550032e+07 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.911744117736816 proc.cpu.threads‚17* disk./.usagePercent‚18.798554388944012 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 197.87890625 disk.drbd1716.in‚0 disk.sda.out‚ 197.87890625 cpu‚25.270675113648643" disk.nvme0c0n1.in‚ 465.83203125 disk.dm-0.out‚43983.63671875 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 465.83203125 disk.sda3.out‚ 197.87890625xèÀ2¥- _wandbruntime‚5225  _runtime‚52256oØãɦà ' lr-AdamW/pg1‚0.00014388026377277295 ' lr-AdamW/pg2‚0.00014388026377277295 trainer/global_step‚5199 % _timestamp‚1.7884332473219213e+09 _runtime‚5225.676947374 _step‚424“¯ÙÎ§È _step‚424 ' lr-AdamW/pg1‚0.00014388026377277295 ' lr-AdamW/pg2‚0.00014388026377277295 trainer/global_step‚5199 % _timestamp‚1.7884332473219213e+09 _runtime‚5225.676947374¨úŒc2¨- _wandbruntime‚5225  _runtime‚5225¾½¯( (©ƒP JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚717497271138304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11112314524672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205166846 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694312205689856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15425441129472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚706388088398848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚603564742809088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚706388088398848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205189958 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111978145 _runtime‚5225.677788942 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚717497271138304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96605817 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚706388062007296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚618990212072960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93211807 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11109209131008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93211807 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111955039 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚717500402923520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40851408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15425388212224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111978145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚706388062007296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40850880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚603564770943488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚705405755417160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111955039 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11090420830412 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11112314524672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55754940 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 trainer/global_step‚5199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55754937 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93211813 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93211813 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 $ _timestamp‚1.788433247322604e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚717500402923520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96606348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚618990131021312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚705402626520268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694312231117824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11109209131008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11093524299336 _step‚425ßzªˆP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚618990131021312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚705402626520268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694312231117824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11109209131008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11093524299336 _step‚425 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚717497271138304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11112314524672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205166846 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694312205689856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15425441129472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚706388088398848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚603564742809088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚706388088398848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205189958 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111978145 _runtime‚5225.677788942 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚717497271138304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96605817 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚706388062007296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚618990212072960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93211807 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11109209131008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93211807 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111955039 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚717500402923520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40851408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15425388212224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytâ{Dð es.small_pool.current‚ 3105393664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111978145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚706388062007296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40850880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚603564770943488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚705405755417160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111955039 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11090420830412 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11112314524672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55754940 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 trainer/global_step‚5199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55754937 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93211813 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93211813 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 $ _timestamp‚1.788433247322604e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚717500402923520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96606348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0©þwµ2«- _wandbruntime‚5226  _runtime‚5226 ^EB'¬ŽN REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40858253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚603677153673216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93229097 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚619105483025920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11111321444864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205228358 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚705537261211684 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11092529623512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111976129 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11114426851840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40858770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205228358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚717634009338880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706519582487040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚619105564064256 trainer/global_step‚5199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93229091 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _runtime‚5226.565063259 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205205220 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96623443 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706519556095488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96623963 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93229091 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚694441602691072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚603677181807616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93229097 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884332482092943e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111976129 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15428382256640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11095633092644 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111999261 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 _step‚426 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11111321444864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706519582487040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552ZÎ1ü'­“N EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93229097 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884332482092943e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111976129 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15428382256640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11095633092644 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111999261 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 _step‚426 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11111321444864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706519582487040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40858253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚603677153673216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93229097 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚619105483025920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11111321444864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205228358 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚705537261211684 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11092529623512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111976129 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11114426851840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40858770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205228358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚717634009338880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706519582487040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚619105564064256 trainer/global_step‚5199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93229091 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _runtime‚5226.565063259 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205205220 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96623443 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706519556095488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96623963 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93229091 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚694441602691072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚603677181807616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302ªbîÀ2®- _wandbruntime‚5226  _runtime‚5226Z={îX¯Ò 0loss/FRt2/assignment_loss‚3.9252819418907166 .loss/FRt2/detection_loss‚2.472277260488934 /loss/FRt4/assignment_loss‚3.524624392390251 0loss/SRqqt2/detection_loss‚2.716719772815704 /loss/FBt3/detection_loss‚2.7513022869825363 /loss/FBt4/detection_loss‚1.9544456195500162 _step‚427 2loss/SRqqt1/assignment_loss‚1.1139434708585574 1loss/FBt2/assignment_loss‚0.13981740137387533 .loss/FRt1/detection_loss‚1.601223382885967 1loss/SRqqt1/detection_loss‚1.0619785444334495 epoch‚4 _runtime‚5226.566129902 /loss/FRt1/assignment_loss‚1.372612512111664 2loss/SRqqt3/assignment_loss‚1.4445446462060014 0loss/SRqqt4/assignment_loss‚1.84043947873371 2loss/FBt1/assignment_loss‚0.059291611149416175 0loss/FBt3/assignment_loss‚0.5083964753089276 /loss/FRt3/detection_loss‚2.4555176306319866 /loss/FBt1/detection_loss‚0.5878700816852821 % _timestamp‚1.7884332482111573e+09 0loss/FBt4/assignment_loss‚0.5192932501021359 0loss/SRqqt3/detection_loss‚1.387954913203915 1loss/SRqqt4/detection_loss‚2.2634975160871234 .loss/FBt2/detection_loss‚2.509734773635864 0loss/FRt3/assignment_loss‚3.0678587712739644 .loss/FRt4/detection_loss‚3.630051113665104 %loss/total_loss‚45.08443282628781 trainer/global_step‚5199 2loss/SRqqt2/assignment_loss‚2.1757559788227083ÎüG]°× .loss/FBt2/detection_loss‚2.509734773635864 0loss/FRt3/assignment_loss‚3.0678587712739644 .loss/FRt4/detection_loss‚3.630051113665104 %loss/total_loss‚45.08443282628781 trainer/global_step‚5199 2loss/SRqqt2/assignment_loss‚2.1757559788227083 0loss/FRt2/assignment_loss‚3.9252819418907166 .loss/FRt2/detection_loss‚2.472277260488934 /loss/FRt4/assignment_loss‚3.524624392390251 0loss/SRqqt2/detection_loss‚2.716719772815704 /loss/FBt3/detection_loss‚2.7513022869825363 /loss/FBt4/detection_loss‚1.9544456195500162 _step‚427 2loss/SRqqt1/assignment_loss‚1.1139434708585574 1loss/FBt2/assignment_loss‚0.13981740137387533 .loss/FRt1/detection_loss‚1.601223382885967 1loss/SRqqt1/detection_loss‚1.0619785444334495 epoch‚4 _runtime‚5226.566129902 /loss/FRt1/assignment_loss‚1.372612512111664 2loss/SRqqt3/assignment_loss‚1.4445446462060014 0loss/SRqqt4/assignment_loss‚1.84043947873371 2loss/FBt1/assignment_loss‚0.059291611149416175 0loss/FBt3/assignment_loss‚0.5083964753089276 /loss/FRt3/detection_loss‚2.4555176306319866 /loss/FBt1/detection_loss‚0.5878700816852821 % _timestamp‚1.7884332482111573e+09 0loss/FBt4/assignment_loss‚0.5192932501021359 0loss/SRqqt3/detection_loss‚1.387954913203915 1loss/SRqqt4/detection_loss‚2.2634975160871234«öÆ ±‚(:’ è¦åÔùýø´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚286.112' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.22240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³lÑ|²‚(:ñ è¦åÔоַ cpu‚25.270049637191057 network.sent‚ 1.2554499e+07( disk.nvme0c0n1.out‚44053.90869140625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0* disk./.usagePercent‚18.801481835098052 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 198.48046875 disk.rbd0.out‚0 disk.rbd1.in‚0$ memory_percent‚88.16831111907959" disk.nvme0c0n1.in‚ 465.83984375 disk.sda3.in‚0.203125 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2934.734375) proc.memory.percent‚17.91219711303711 proc.cpu.threads‚17) proc.memory.availableMB‚ 1938.50390625 disk.nvme0n1.in‚ 465.83984375% disk./.usageGB‚280.15566635131836 disk.dm-0.in‚ 465.83984375# disk.dm-0.out‚44053.90869140625& disk.nvme0n1.out‚44053.90869140625 disk.sda2.in‚0 disk.dm-1.out‚ 198.48046875 disk.sda3.out‚ 198.48046875 network.recv‚ 1.846484e+06 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.759765625œ<ù“³‚(:ˆ ÷¦åÔ·µ÷´ gpu.0.gpu‚77.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚160.546' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L£)¤~´‚(:ó ÷¦åÔ·¾£¸ disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 cpu‚25.269417950216308 network.sent‚ 1.2660425e+07 disk.nvme0n1.in‚ 465.83984375 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0$ memory_percent‚88.16876411437988# proc.memory.rssMB‚ 2934.85546875 network.recv‚ 1.851563e+06* disk./.usagePercent‚18.804462274811677 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0% disk./.usageGB‚280.20007705688477& disk.nvme0n1.out‚44098.78759765625 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 199.16015625 disk.dm-1.out‚ 199.16015625" disk.nvme0c0n1.in‚ 465.83984375( disk.nvme0c0n1.out‚44098.78759765625 disk.drbd1716.out‚0* proc.memory.percent‚17.912936210632324 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 2.759765625 disk.dm-0.in‚ 465.83984375 disk.sda3.in‚0.203125 disk.rbd0.in‚0# disk.dm-0.out‚44098.78759765625 disk.drbd2080.in‚0( proc.memory.availableMB‚ 1938.4296875 disk.sda3.out‚ 199.16015625RCgµ‚(:’ †§åÔ»Îý´ gpu.0.gpu‚61.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚50.0 gpu.0.powerWatts‚307.708' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚61.54160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P¾ût¶‚(:é †§åÔÕÓï¶ disk.rbd1.in‚0 disk.dm-0.in‚ 465.83984375 disk.sda3.in‚0.203125 proc.memory.rssMB‚ 2934.90625 network.sent‚ 1.2664443e+07* proc.memory.percent‚17.913246154785156 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.rbd1.out‚ 2.759765625) proc.memory.availableMB‚ 1938.55859375 network.recv‚ 1.853623e+06$ disk./.usageGB‚280.2174873352051 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚44182.4755859375 disk.sda.out‚ 199.6015625$ memory_percent‚88.16797733306885 disk.rbd0.out‚0 disk.nvme0n1.in‚ 465.83984375% disk.nvme0n1.out‚44182.4755859375' disk.nvme0c0n1.out‚44182.4755859375 disk.drbd1716.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚18.80563069319795" disk.nvme0c0n1.in‚ 465.83984375 disk.sda3.out‚ 199.6015625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.268715275779478 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 199.6015625 disk.drbd2266.out‚02,ÊÏ2·- _wandbruntime‚5272  _runtime‚5272VY{úɸà % _timestamp‚1.7884332937758746e+09 _runtime‚5272.130913292 _step‚428 ' lr-AdamW/pg1‚0.00014252697993042418 ' lr-AdamW/pg2‚0.00014252697993042418 trainer/global_step‚5249cØLÎ¹È ' lr-AdamW/pg2‚0.00014252697993042418 trainer/global_step‚5249 % _timestamp‚1.7884332937758746e+09 _runtime‚5272.130913292 _step‚428 ' lr-AdamW/pg1‚0.00014252697993042418¬°Ø†±2º- _wandbruntime‚5272  _runtime‚5272[D8ÿ»„P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94076013 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113035997 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11217944631296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚609185314149888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚711981058724168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113012891 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.largã;§Ãô#e_pool.allocated‚56267590 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚700782055750656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚724180737439744 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11198977545544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 % _timestamp‚1.7884332937765906e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207088898 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97497697 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚724177605654528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11217944631296 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5272.131873434 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15572741640192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11214839237632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚429 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚712962792808448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚724177605654528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15572686625792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11195874076620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚712962766416896 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚624758055790080 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94076013 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113012891 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚711977929827276 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚712962792808448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41230646 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚724180737439744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚712962766416896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94076007 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56267587 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚624757972641280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113035997 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚712962792808448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚724177605654528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15572686625792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11195874076620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚712962766416896 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚624758055790080 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94076013 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113012891 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚711977929827276 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚712962792808448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41230646 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚724180737439744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚712962766416896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94076007 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56267587 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚624757972641280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113035997 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97497697 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚724177605654528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11217944631296 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5272.131873434 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800­›Œ¹2½- _wandbruntime‚5273  _runtime‚5273Ô"§É'¾ŽN ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚700911452751872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚700911478179840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚5249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113057113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113033981 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713094286896640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97516182 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚712112564518692 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚609297696879616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207127272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚724311212056576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 _runtime‚5273.023326328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624873413550592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15575633535488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94093291 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11197982869720 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207150410 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713094260505088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207127272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94093297 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97515668 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11201086338852 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56277843 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624873330415104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113033981 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚713094286896640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11216951551488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56277840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11220056958464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113057113 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚724314343855104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207150410º*=@” ¿“N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚609297696879616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207127272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚724311212056576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 _runtime‚5273.023326328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624873413550592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15575633535488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94093291 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11197982869720 ?6DeviceStatsMonitor.on_train_batch_enddk™€…/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207150410 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713094260505088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207127272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94093297 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97515668 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11201086338852 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56277843 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624873330415104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113033981 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚713094286896640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11216951551488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56277840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11220056958464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113057113 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚724314343855104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207150410 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚700911452751872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚700911478179840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚5249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113057113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113033981 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713094286896640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97516182 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚712112564518692 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878®èÿº2À- _wandbruntime‚5273  _runtime‚5273ZüQ#VÁÐ .loss/FRt2/detection_loss‚2.451896125078201 /loss/FBt2/detection_loss‚2.7174543380737304 .loss/FRt3/assignment_loss‚2.97345488912919 2loss/SRqqt2/assignment_loss‚1.8062955503528182 2loss/SRqqt4/assignment_loss‚2.2121746383003287 /loss/FRt4/detection_loss‚3.1414025936807906 trainer/global_step‚5249 2loss/SRqqt1/assignment_loss‚1.1813033779754358 1loss/SRqqt3/assignment_loss‚1.365652418486318 0loss/FBt1/assignment_loss‚0.0758851492277809 /loss/FBt2/assignment_loss‚0.319988943637685 0loss/SRqqt1/detection_loss‚1.012278954947696 1loss/SRqqt3/detection_loss‚1.8971165972856843 1loss/SRqqt2/detection_loss‚2.2773735192981928 0loss/SRqqt4/detection_loss‚2.340841723454965 /loss/FBt1/detection_loss‚0.9152987842699365 .loss/FBt3/detection_loss‚2.925232912812914 _runtime‚5273.024843742 /loss/FRt1/detection_loss‚1.4646980843347748 0loss/FRt1/assignment_loss‚1.8552683875328158 0loss/FRt4/assignment_loss‚3.9770802472318922 % _timestamp‚1.7884332946698868e+09 _step‚431 1loss/FBt3/assignment_loss‚0.18784191407861986 -loss/FRt3/detection_loss‚2.47183267969419 .loss/FBt4/detection_loss‚1.611545111003675 0loss/FRt2/assignment_loss‚3.6612565591931343 1loss/FBt4/assignment_loss‚0.31772187488035053 %loss/total_loss‚45.16089537396112 epoch‚4þxŠI[ÂÕ 2loss/SRqqt2/assignment_loss‚1.8062955503528182 2loss/SRqqt4/assignment_loss‚2.2121746383003287 /loss/FRt4/detection_loss‚3.1414025936807906 trainer/global_step‚5249 2loss/SRqqt1/assignment_loss‚1.1813033779754358 1loss/SRqqt3/assignment_loss‚1.365652418486318 0loss/FBt1/assignment_loss‚0.0758851492277809 /loss/FBt2/assignment_loss‚0.319988943637685 0loss/SRqqt1/detection_loss‚1.012278954947696 1loss/SRqqt3/detection_loss‚1.8971165972856843 1loss/SRqqt2/detection_loss‚2.2773735192981928 0loss/SRqqt4/detection_loss‚2.340841723454965 /loss/FBt1/detection_loss‚0.9152987842699365 .loss/FBt3/detection_loss‚2.925232912812914 _runtime‚5273.024843742 /loss/FRt1/detection_loss‚1.4646980843347748 0loss/FRt1/assignment_loss‚1.8552683875328158 0loss/FRt4/assignment_loss‚3.9770802472318922 % _timestamp‚1.7884332946698868e+09 _step‚431 1loss/FBt3/assignment_loss‚0.18784191407861986 -loss/FRt3/detection_loss‚2.47183267969419 .loss/FBt4/detection_loss‚1.611545111003675 0loss/FRt2/assignment_loss‚3.6612565591931343 1loss/FBt4/assignment_loss‚0.31772187488035053 %loss/total_loss‚45.16089537396112 epoch‚4 .loss/FRt2/detection_loss‚2.451896125078201 /loss/FBt2/detection_loss‚2.7174543380737304 .loss/FRt3/assignment_loss‚2.97345488912919¯Å7-tÂ(:é •§åÔ㟳· proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 465.83984375 disk.sda3.out‚ 200.2734375 disk.sda.in‚0.203125" disk.nvme0c0n1.in‚ 465.83984375) proc.memory.availableMB‚ 1938.24609375 disk.dm-1.in‚0.203125 network.sent‚ 1.2669255e+07) disk./.usagePercent‚18.80806199675063 disk.sda3.in‚0.203125 disk.sda.out‚ 200.2734375 disk.rbd1.in‚0$ memory_percent‚88.16988468170166! proc.memory.rssMB‚ 2934.953125" disk.dm-0.out‚44257.3076171875 disk.drbd2265.out‚0% disk.nvme0n1.out‚44257.3076171875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 465.83984375 cpu‚25.26809168474624 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.759765625' disk.nvme0c0n1.out‚44257.3076171875* proc.memory.percent‚17.913532257080078 disk.dm-1.out‚ 200.2734375 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.recv‚ 1.856956e+06$ disk./.usageGB‚280.2537155151367 disk.drbd2265.in‚0 disk.rbd0.in‚0¹u“Ä‚(:ˆ •§åÔé³íØ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚149.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{YàžÅ‚(:“ ¤§åÔ‡·¹µ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚138.948' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.789600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H…JjÆ‚(:ß ¤§åÔÓ¶á·" disk.nvme0n1.out‚ 44326.8984375 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 200.7734375 disk.dm-0.out‚ 44326.8984375 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.sda.out‚ 200.7734375 disk.drbd2266.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚17.913532257080078 proc.cpu.threads‚17 disk.sda.in‚0.203125 network.sent‚ 1.2777277e+07 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 44326.8984375 disk.dm-0.in‚ 465.83984375 disk.drbd2080.in‚0 disk.rbd0.out‚0 cpu‚25.267473381105837 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 network.recv‚ 1.863267e+06) disk./.usagePercent‚18.80865491019993$ memory_percent‚88.16862106323242 disk.nvme0n1.in‚ 465.83984375 disk.drbd2080.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1938.453125! proc.memory.rssMB‚ 2934.953125" disk.nvme0c0n1.in‚ 465.83984375 disk.sda3.out‚ 200.7734375 disk.drbd1716.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚280.2625503540039汓¹Ç‚(:’ ³§åÔÿƧµ gpu.0.gpu‚94.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚265.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.13680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óGÛmxÈ‚(:í ³§åÔ«¸¢· network.sent‚ 1.2781434e+07* disk./.usagePercent‚18.810953857857406 disk.dm-1.out‚ 201.47265625 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2934.953125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.rbd0.out‚0$ memory_percent‚88.16859722137451" disk.dm-0.out‚44394.5068359375 disk.drbd1716.in‚0 disk.rbd0.in‚0 cpu‚25.266819889521244 disk.nvme0n1.in‚ 465.83984375% disk.nvme0n1.out‚44394.5068359375" disk.nvme0c0n1.in‚ 465.83984375 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1938.45703125$ disk./.usageGB‚280.2968063354492 disk.sda2.out‚0' disk.nvme0c0n1.out‚44394.5068359375 disk.drbd1716.out‚0 disk.drbd2265.in‚0* proc.memory.percent‚17.913532257080078 network.recv‚ 1.86579e+06 disk.dm-0.in‚ 465.83984375 disk.sda3.out‚ 201.47265625 disk.sda.out‚ 201.47265625¢S‡2É- _wandbruntime‚5318  _runtime‚5318|‚¿ÉÊà _runtime‚5318.940520772 _step‚432 ' lr-AdamW/pg1‚0.00014116530976422573 ' lr-AdamW/pg2‚0.00014116530976422573 trainer/global_step‚5299 % _timestamp‚1.7884333405854378e+09ÂVXÎËÈ trainer/global_step‚5299 % _timestamp‚1.7884333405854378e+09 _runtime‚5318.940520772 _step‚432 ' lr-AdamW/pg1‚0.00014116530976422573 ' lr-AdamW/pg2‚0.00014116530976422573°>Ê$X2Ì- _wandbruntime‚5318  _runtime‚5318§û Ò (Í„P H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _step‚433 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 trainer/global_step‚5299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94940213 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚730857940170752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56780240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630525651820544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15719875515904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114093849 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚719537470826496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94940213 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚730857940170752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚719537470826496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11320469344256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11320469344256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209034062 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114093849 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98391080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630525732872192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚730861071955968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚718556362031176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚707251931239424 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11304430791752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94940207 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94940207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520Þ'3 ÿΉP WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚719537470826496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11320469344256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11320469344256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209034062 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114093849 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98391080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630525732872192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚730861071955968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚718556362031176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚707251931239424 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11304430791752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94940207 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94940207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _step‚433 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 trainer/global_step‚5299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94940213 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚730857940170752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56780240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630525651820544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15719875515904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114093849 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚719537470826496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94940213 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚730857940170752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878±š¶Ý2Ï- _wandbruntime‚5319  _runtime‚5319*„ºŸ'ÐŽN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209072462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94957497 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56790490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚630641000947712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚718687867825700 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _runtime‚5319.829752593 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11322581658112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94957491 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11303436115928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94957491 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11325687065088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15722760861696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11306539585060 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209049324 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614918268220416 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091833 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚730994678371328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚707381302812672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41618917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41618396 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚730991546572800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11325687065088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94957497 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 % _timestamp‚1.7884333414739287e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15722815862784 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 _step‚434 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98408886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚719668991306240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114114965 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚718684738928600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209072462 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114091833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚614918240086016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚719668991306240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98409410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56790493 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚707381328240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0@C=¿'Ñ“N PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98409410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56790493 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚707381328240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209072462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94957497 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56790490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚630641000947712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚718687867825700 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _runtime‚5319.829752593 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11322581658112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94957491 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11303436115928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94957491 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11325687065088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15722760861696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11306539585060 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209049324 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614918268220416 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091833 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚730994678371328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚707381302812672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41618917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41618396 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚730991546572800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11325687065088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94957497 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 % _timestamp‚1.7884333414739287e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15722815862784 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 _step‚434 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98408886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚719668991306240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114114965 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚718684738928600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209072462 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114091833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚614918240086016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚719668991306240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FkÚ‚(:à à§åԼ☸ network.sent‚ 1.2896143e+07* disk./.usagePercent‚18.814615021604578% disk.nvme0c0n1.out‚44631.10546875 disk.rbd1.in‚0* proc.memory.percent‚17.914247512817383 disk.drbd2080.out‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚44631.10546875 disk.dm-1.in‚0.203125$ disk./.usageGB‚280.3513603210449 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 network.recv‚ 1.875368e+06 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚25.26495126309182 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 466.7265625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.dm-0.out‚44631.10546875 disk.sda1.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 202.9296875 disk.drbd2265.in‚0 disk.rbd1.out‚ 2.759765625$ memory_percent‚88.16957473754883" proc.memory.rssMB‚ 2935.0703125 disk.sda3.out‚ 202.9296875 disk.drbd1716.out‚0 disk.dm-1.out‚ 202.9296875' proc.memory.availableMB‚ 1938.296875 disk.dm-0.in‚ 466.7265625 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.7265625 disk.drbd2266.in‚07½·42Û- _wandbruntime‚5365  _runtime‚5365$øÿÈÜ _runtime‚5365.603504956 _step‚436 ' lr-AdamW/pg1‚0.00013979553704419927 ' lr-AdamW/pg2‚0.00013979553704419927 trainer/global_step‚5349 $ _timestamp‚1.788433387248451e+09²1™ÈÍÝÇ ' lr-AdamW/pg1‚0.00013979553704419927 ' lr-AdamW/pg2‚0.00013979553704419927 trainer/global_step‚5349 $ _timestamp‚1.788433387248451e+09 _runtime‚5365.603504956 _step‚436´«$ë2Þ- _wandbruntime‚5365  _runtime‚5365 aì¥'߃P L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95804407 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11429204844544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115128595 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15866748922368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚726112175236096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11426099450880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41989601 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620426372428288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚737538274686976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99282491 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210933002 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115128595 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95804407 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115151701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41989051 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚725128536441292 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115151701 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚737538274686976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚726112175236096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11429204844544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚737541406472192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚726112201627648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210956114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57292890 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _step‚437 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11426099450880 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11406780569036 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636293121350656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚713721781300224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95804413 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99281938 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 % _timestamp‚1.7884333872492301e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚713721755872256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚726112201627648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984'Mê™(àˆP VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚737541406472192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57292887 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99281938 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 % _timestamp‚1.7884333872492301e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚713721755872256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚726112201627648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95804407 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11429204844544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115128595 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15866748922368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚726112175236096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11426099450880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41989601 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620426372428288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚737538274686976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99282491 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210933002 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115128595 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95804407 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115151701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41989051 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚725128536441292 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115151701 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚737538274686976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚726112175236096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11429204844544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚737541406472192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚726112201627648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210956114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57292890 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _step‚437 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11426099450880 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11406780569036 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636293121350656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚713721781300224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95804413 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 trainer/global_step‚5349 _step‚438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620538783292416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11428211764736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41997017 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210971376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95821697 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210994514 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57303140 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210971376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620538811426816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚725260042235608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11408889362136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11428211764736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99299627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚636408535986176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115172817 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737671881089024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95821697 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41996487 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636408450753536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15869667461120 $ _timestamp‚1.788433388134448e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚726243695715840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚726243669324288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 FDeviceStatsMonitor.on_train_bUôpatch_end/allocated_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 trainer/global_step‚5349 _step‚438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620538783292416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11428211764736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41997017 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210971376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95821697 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210994514 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57303140 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210971376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620538811426816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚725260042235608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11408889362136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11428211764736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99299627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚636408535986176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115172817 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737671881089024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95821697 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41996487 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636408450753536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15869667461120 $ _timestamp‚1.788433388134448e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚726243695715840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚726243669324288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16013393986560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116209553 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731703839748300 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116186447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42369976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642060394867200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11534834951168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57805540 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96668613 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626046915634688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96668613 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11531729557504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚720191631361024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11515337284168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11531729557504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744218609203200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚744218609203200 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116209553 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚732686906037248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42370533 _runtime‚5412.100323083 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚744221740988416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100176073 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212878166 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626046943769088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚732686879645696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚720191605933056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440›I˜6"òŠP M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚731706968645192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116186447 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642060394867200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11534834951168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57805540 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96668613 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626046915634688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96668613 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11531729557504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚720191631361024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11515337284168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11531729557504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744218609203200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚744218609203200 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116209553 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚732686906037248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42370533 _runtime‚5412.100323083 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚744221740988416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100176073 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212878166 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626046943769088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚732686879645696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚720191605933056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitžxÚor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 $ _timestamp‚1.788433433745065e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16013393986560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116209553 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731703839748300 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116186447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42369976 B¦ê'ôN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96685897 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42377528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚731838474439716 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642175734964736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96685891 _step‚442 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚720321028362240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100193318 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212893428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5412.990254144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16016323233280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732818400125440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96685891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57815793 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚744355347403776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116230669 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11514342608344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚731835345542616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116207537 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732818400125440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11533841871360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚732818373733888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212893428 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11536947278336 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116207537 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642175649732096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884334346348329e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744352215605248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212916566 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96685897 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16016380331520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212916566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 Ûg'õ•N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744352215605248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212916566 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96685897 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16016380331520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212916566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96685897 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42377528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚731838474439716 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642175734964736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96685891 _step‚442 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚720321028362240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100193318 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212893428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5412.990254144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16016323233280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732818400125440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96685891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57815793 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚744355347403776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116230669 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11514342608344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚731835345542616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116207537 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732818400125440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11533841871360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚732818373733888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212893428 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11536947278336 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116207537 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642175649732096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884334346348329e+09ºdüå™2ö- _wandbruntime‚5412  _runtime‚5412:(¸¾V÷Ð /loss/FRt4/detection_loss‚2.9945602152082653 trainer/global_step‚5399 /loss/FRt1/detection_loss‚1.6603886524184805 /loss/FRt4/assignment_loss‚4.003216363765575 0loss/FBt1/assignment_loss‚0.1490599384497824 2loss/SRqqt2/assignment_loss‚2.3702204103271165 0loss/FBt2/assignment_loss‚0.5062520278318386 1loss/SRqqt3/detection_loss‚1.3969213131447913 .loss/FBt2/detection_loss‚2.274431586265564 /loss/FRt2/detection_loss‚2.9662755340337754 /loss/FBt1/detection_loss‚0.8160268581871476 .loss/FBt4/detection_loss‚1.576170668218817 /loss/FBt4/assignment_loss‚0.446230546440555 /loss/FRt2/assignment_loss‚4.045463728904724 0loss/FRt3/assignment_loss‚2.9227811971376108 1loss/SRqqt3/assignment_loss‚1.267257598189401 0loss/FBt3/assignment_loss‚0.3938936699226949 1loss/SRqqt1/assignment_loss‚1.314159457603919 1loss/SRqqt2/detection_loss‚3.2559845334007624 0loss/FRt1/assignment_loss‚1.4612747821651522 2loss/SRqqt4/assignment_loss‚1.9729069820765792 .loss/FRt3/detection_loss‚2.423873853544856 1loss/SRqqt1/detection_loss‚1.0523906536828513 _runtime‚5412.991749386 _step‚443 0loss/SRqqt4/detection_loss‚2.554897499495539 /loss/FBt3/detection_loss‚1.9668332479502026 %loss/total_loss‚45.79147131836601 epoch‚4 % _timestamp‚1.7884334346364636e+09L'Á'[øÕ /loss/FRt2/assignment_loss‚4.045463728904724 0loss/FRt3/assignment_loss‚2.9227811971376108 1loss/SRqqt3/assignment_loss‚1.267257598189401 0loss/FBt3/assignment_loss‚0.3938936699226949 1loss/SRqqt1/assignment_loss‚1.314159457603919 1loss/SRqqt2/detection_loss‚3.2559845334007624 0loss/FRt1/assignment_loss‚1.4612747821651522 2loss/SRqqt4/assignment_loss‚1.9729069820765792 .loss/FRt3/detection_loss‚2.423873853544856 1loss/SRqqt1/detection_loss‚1.0523906536828513 _runtime‚5412.991749386 _step‚443 0loss/SRqqt4/detection_loss‚2.554897499495539 /loss/FBt3/detection_loss‚1.9668332479502026 %loss/total_loss‚45.79147131836601 epoch‚4 % _timestamp‚1.7884334346364636e+09 /loss/FRt4/detection_loss‚2.9945602152082653 trainer/global_step‚5399 /loss/FRt1/detection_loss‚1.6603886524184805 /loss/FRt4/assignment_loss‚4.003216363765575 0loss/FBt1/assignment_loss‚0.1490599384497824 2loss/SRqqt2/assignment_loss‚2.3702204103271165 0loss/FBt2/assignment_loss‚0.5062520278318386 1loss/SRqqt3/detection_loss‚1.3969213131447913 .loss/FBt2/detection_loss‚2.274431586265564 /loss/FRt2/detection_loss‚2.9662755340337754 /loss/FBt1/detection_loss‚0.8160268581871476 .loss/FBt4/detection_loss‚1.576170668218817 /loss/FBt4/assignment_loss‚0.446230546440555»[ â“ù‚(:ˆ œ¨åÔ™Âѵ gpu.0.gpu‚81.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚126.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Ãfú‚(:Û œ¨åÔ¤’¸ proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 466.734375% disk.nvme0c0n1.out‚45120.98828125 disk.dm-0.in‚ 466.734375 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0$ memory_percent‚88.17260265350342) proc.memory.availableMB‚ 1937.80078125 cpu‚25.26248923313002 disk.sda3.out‚ 205.0234375 disk.sda.out‚ 205.0234375 disk.rbd0.out‚0$ disk./.usageGB‚280.5343437194824# disk.nvme0n1.out‚45120.98828125 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 205.0234375 disk.drbd2266.in‚0 network.recv‚ 1.891418e+06 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.3030996e+07 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚17.915725708007812) disk./.usagePercent‚18.82689519100702 disk.nvme0n1.in‚ 466.734375 disk.dm-0.out‚45120.98828125 proc.memory.rssMB‚ 2935.3125¶㵓û‚(:ˆ «¨åÔ……Ó´ gpu.0.gpu‚75.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚130.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Y Wü‚(:Ì «¨åÔÝ‘¶¸ disk.drbd2266.in‚0 disk.dm-0.out‚ 45233.203125 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.734375 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.734375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.recv‚ 1.8985e+06* disk./.usagePercent‚18.829654184203523 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 466.734375 disk.sda.out‚205.5625$ memory_percent‚88.17124366760254 disk.rbd0.in‚0 disk.dm-1.out‚205.5625 disk.rbd1.out‚ 2.759765625% disk./.usageGB‚280.57545471191406! disk.nvme0n1.out‚ 45233.203125 disk.sda3.in‚0.203125( proc.memory.availableMB‚ 1938.0234375 proc.memory.rssMB‚ 2935.3125 disk.sda2.in‚0 disk.sda3.out‚205.5625* proc.memory.percent‚17.915725708007812 cpu‚25.261902390825107 network.sent‚ 1.3126555e+07# disk.nvme0c0n1.out‚ 45233.203125 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0ûŒý‚(: º¨åÔ÷ûÁµ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚202.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.5 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðцýpþ‚(:å º¨åÔŒöí· disk.drbd2265.in‚0$ memory_percent‚88.17133903503418 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 206.0859375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 2.759765625 disk.nvme0c0n1.in‚ 466.734375 disk.sda.in‚0.203125% disk.nvme0n1.out‚45385.4384765625 disk.dm-1.in‚0.203125 disk.rbd1.in‚0* proc.memory.percent‚17.915773391723633 network.sent‚ 1.3132298e+07 network.recv‚ 1.901932e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.734375 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚45385.4384765625 disk.dm-0.in‚ 466.734375" disk.dm-0.out‚45385.4384765625( proc.memory.availableMB‚ 1938.0078125" proc.memory.rssMB‚ 2935.3203125 cpu‚25.261280981062374) disk./.usagePercent‚18.83184688873429% disk./.usageGB‚280.60812759399414 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 206.0859375 disk.sda2.in‚0 disk.dm-1.out‚ 206.0859375ö6v2ÿ- _wandbruntime‚5458  _runtime‚5458LžâØÇ€Á & lr-AdamW/pg1‚0.0001370328274060605 & lr-AdamW/pg2‚0.0001370328274060605 trainer/global_step‚5449 % _timestamp‚1.7884334805003858e+09 _runtime‚5458.855442346 _step‚444œ#ÜÌÆ & lr-AdamW/pg1‚0.0001370328274060605 & lr-AdamW/pg2‚0.0001370328274060605 trainer/global_step‚5449 % _timestamp‚1.7884334805003858e+09 _runtime‚5458.855442346 _step‚444¼áîL32‚- _wandbruntime‚5458  _runtime‚5458$b;·Œ ƒ„P `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16160056197120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11617687061452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214777106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42750391 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16160113308672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚726661455993856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647827515038208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101068581 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚750898943719424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes¦,~.all.allocated‚750902075504640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚739261584055296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214800218 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117267405 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11640465057792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11637359664128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42749826 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738282271952200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚726661481421824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11637359664128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117267405 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚739261584055296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58318190 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _runtime‚ 5458.85629225 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚739261610446848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117244299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329¾¾ƒû(„‰P H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101068581 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚750898943719424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚750902075504640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚739261584055296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214800218 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117267405 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11640465057792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11637359664128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42749826 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738282271952200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚726661481421824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11637359664128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117267405 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚739261584055296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58318190 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _runtime‚ 5458.85629225 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚739261610446848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117244299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16160056197120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11617687061452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214777106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42750391 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16160113308672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚726661455993856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647827515038208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647942856001024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214815480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42757787 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚739393104535040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738410648849624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97550097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117265389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58328440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11619795854552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214838618 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚5449 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101085690 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11622899323684 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚5459.751115618 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97550091 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚751035681920000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11642577384960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97550097 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚751032550121472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚739393078143488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58328443 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16162986295808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42757250‰ž¥Ÿ ‡“N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11639471977984 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101085690 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11622899323684 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚5459.751115618 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97550091 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚751035681920000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11642577384960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97550097 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚751032550121472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚739393078143488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58328443 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16162986295808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42757250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pooP³ml.peak‚1878 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647942856001024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214815480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42757787 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚739393104535040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738410648849624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97550097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117265389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58328440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11619795854552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214838618 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚5449 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757579278235648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16306728156672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745836314856448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11746095164416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216699158 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101963203 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58830837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653594811255808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚744854446362316 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11726243776584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757582410020864 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58830840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚5499 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _runtime‚5505.640699382 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98397007 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757579278235648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745836288464896 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0xÓ’Ýf ˜ŠP KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745836314856448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11746095164416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216699158 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101963203 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58830837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653594811255808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚744854446362316 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11726243776584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757582410020864 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58830840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚5499 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _runtime‚5505.640699382 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98397007 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757579278235648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745836288464896 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757579278235648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16306728156672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733260703055872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118323241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98414297 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43139937 _step‚450 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚745967782553088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98414297 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚744989081053732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚745967808944640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216760670 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216737532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101981027 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101981557 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16309659400704 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43140464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884335281783648e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11748207491584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚637400441046016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11748207491584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118346373 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216760670 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 trainer/global_step‚5499§Ñ‘'›•N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216760670 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 trainer/global_step‚5499 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11745102084608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58841093 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚744985952156632 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11725249100760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98414291 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚745967808944640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733260703055872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118323241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98414297 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43139937 _step‚450 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚745967782553088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98414297 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚744989081053732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚745967808944640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216760670 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216737532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101981027 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101981557 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16309659400704 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43140464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884335281783648e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11748207491584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚637400441046016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11748207491584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118346373 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180ª؇2œ- _wandbruntime‚5506  _runtime‚5506“(ݾ[Õ 0loss/FRt3/assignment_loss‚1.9434256381458705 /loss/FRt2/detection_loss‚2.9108302742242813 1loss/SRqqt4/detection_loss‚3.1100978062433353 2loss/SRqqt3/assignment_loss‚1.1751245689726826 /loss/FBt2/detection_loss‚2.4854540791776447 $loss/total_loss‚44.5630977756037 _step‚451 /loss/FRt1/assignment_loss‚2.066959948973222 /loss/FRt4/assignment_loss‚4.049019006582407 2loss/SRqqt2/assignment_loss‚1.9623269867897033 .loss/FBt3/detection_loss‚2.352210776011149 epoch‚4 1loss/SRqqt3/detection_loss‚1.3351745294487996 /loss/FRt2/assignment_loss‚4.098855090992791 1loss/SRqqt4/assignment_loss‚1.639814770397018 1loss/FBt2/assignment_loss‚0.19578796525446604 /loss/FBt4/detection_loss‚1.6245738478998344 _runtime‚5506.535563206 2loss/SRqqt1/assignment_loss‚1.1870648307701597 .loss/FRt3/detection_loss‚2.188783868485027 /loss/FRt4/detection_loss‚2.9845645977900577 3loss/FBt1/assignment_loss‚0.0037267656857767526 0loss/FBt3/assignment_loss‚0.2737438186731197 0loss/FBt4/assignment_loss‚0.1818379308679141 /loss/FRt1/detection_loss‚1.9175784220072356 /loss/FBt1/detection_loss‚1.0907407118938863 trainer/global_step‚5499 % _timestamp‚1.7884335281805134e+09 0loss/SRqqt1/detection_loss‚1.036057680167935 1loss/SRqqt2/detection_loss‚2.7493438601493834oÕÃ`žÚ trainer/global_step‚5499 % _timestamp‚1.7884335281805134e+09 0loss/SRqqt1/detection_loss‚1.036057680167935 1loss/SRqqt2/detection_loss‚2.7493438601493834 0loss/FRt3/assignment_loss‚1.9434256381458705 /loss/FRt2/detection_loss‚2.9108302742242813 1loss/SRqqt4/detection_loss‚3.1100978062433353 2loss/SRqqt3/assignment_loss‚1.1751245689726826 /loss/FBt2/detection_loss‚2.4854540791776447 $loss/total_loss‚44.5630977756037 _step‚451 /loss/FRt1/assignment_loss‚2.066959948973222 /loss/FRt4/assignment_loss‚4.049019006582407 2loss/SRqqt2/assignment_loss‚1.9623269867897033 .loss/FBt3/detection_loss‚2.352210776011149 epoch‚4 1loss/SRqqt3/detection_loss‚1.3351745294487996 /loss/FRt2/assignment_loss‚4.098855090992791 1loss/SRqqt4/assignment_loss‚1.639814770397018 1loss/FBt2/assignment_loss‚0.19578796525446604 /loss/FBt4/detection_loss‚1.6245738478998344 _runtime‚5506.535563206 2loss/SRqqt1/assignment_loss‚1.1870648307701597 .loss/FRt3/detection_loss‚2.188783868485027 /loss/FRt4/detection_loss‚2.9845645977900577 3loss/FBt1/assignment_loss‚0.0037267656857767526 0loss/FBt3/assignment_loss‚0.2737438186731197 0loss/FBt4/assignment_loss‚0.1818379308679141 /loss/FRt1/detection_loss‚1.9175784220072356 /loss/FBt1/detection_loss‚1.0907407118938863ÃOE`Å“Ÿ‚(:ˆ …©åÔׂµ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.9778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²²²…o ‚(:ä …©åÔ°´µ· disk.drbd1716.out‚0 disk.drbd2266.in‚0$ memory_percent‚88.17403316497803) proc.memory.availableMB‚ 1937.56640625* disk./.usagePercent‚18.840405220603945( disk.nvme0c0n1.out‚45850.37744140625 disk.sda3.in‚0.203125 disk.sda3.out‚ 209.046875 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2935.421875 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 209.046875 disk.dm-1.out‚ 209.046875 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.734375# disk.dm-0.out‚45850.37744140625 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.734375 cpu‚25.258337713451787& disk.nvme0n1.out‚45850.37744140625 network.recv‚ 1.92362e+06 disk.nvme0c0n1.in‚ 466.734375 network.sent‚ 1.3260416e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 2.9404296875 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0* proc.memory.percent‚17.916393280029297# disk./.usageGB‚280.735652923584±(¹ž¡‚(:“ ”©åÔëûµ gpu.0.gpu‚81.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.276799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 âxÍs¢‚(:è ”©åÔù¤¤¸ disk.dm-0.in‚ 466.734375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚ 1.929122e+06# memory_percent‚88.1725549697876) proc.memory.availableMB‚ 1937.80859375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 209.47265625 disk.rbd0.out‚0% disk.nvme0n1.out‚45944.1416015625 disk.sda2.in‚0 disk.drbd2266.out‚0" disk.dm-0.out‚45944.1416015625 disk.sda3.in‚0.203125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.734375 disk.dm-1.out‚ 209.47265625 disk.drbd2266.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚18.842522146904578% disk./.usageGB‚280.76719665527344 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.nvme0c0n1.in‚ 466.734375* proc.memory.percent‚17.916393280029297 cpu‚25.25781246768937 disk.sda3.out‚ 209.47265625 disk.sda1.out‚0 disk.rbd1.out‚ 2.9404296875' disk.nvme0c0n1.out‚45944.1416015625! proc.memory.rssMB‚ 2935.421875 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3368202e+07 disk.drbd2080.out‚0•¡ËÌ“£‚(:ˆ £©åÔíÛÙµ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚299.624' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ASRs¤‚(:è £©åÔƹµ¶ disk.rbd0.out‚0 cpu‚25.257226235942408* disk./.usagePercent‚18.845306484846976 disk.sda3.out‚ 210.16015625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.00341796875 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda.out‚ 210.16015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0.203125 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 466.734375# memory_percent‚88.1725549697876! proc.memory.rssMB‚ 2935.421875 disk.sda1.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚46038.5322265625$ disk./.usageGB‚280.8086853027344 disk.nvme0n1.in‚ 466.734375 disk.dm-1.out‚ 210.16015625) proc.memory.availableMB‚ 1937.80859375* proc.memory.percent‚17.916393280029297 network.sent‚ 1.337249e+07" disk.dm-0.out‚46038.5322265625 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2265.in‚0% disk.nvme0n1.out‚46038.5322265625 network.recv‚ 1.932629e+06 disk.dm-0.in‚ 466.734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0vÖb2¥- _wandbruntime‚5552  _runtime‚5552›Î×ɦà _runtime‚5552.177173814 _step‚452 ' lr-AdamW/pg1‚0.00013424115387421456 ' lr-AdamW/pg2‚0.00013424115387421456 trainer/global_step‚5549 % _timestamp‚1.7884335738221362e+09Gôl¤Î§È % _timestamp‚1.7884335738221362e+09 _runtime‚5552.177173814 _step‚452 ' lr-AdamW/pg1‚0.00013424115387421456 ' lr-AdamW/pg2‚0.00013424115387421456 trainer/global_step‚5549Äô"N2¨- _wandbruntime‚5552  _runtime‚55529ÎßMt ©†P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752410992874496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16453205854208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚739601156115456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11831697022792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43510209 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218644322 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752411019266048 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11851725271040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751432878566216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119360003 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764259612751872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752410992874496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚659361779242496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚739601181543424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119383109 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764259612751872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764262744537088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119383109 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102854238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚5552.178187086 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764262744537088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99261207 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 38766930965Øz(ª‹P VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752410992874496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16453205854208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚739601156115456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11831697022792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43510209 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218644322 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752411019266048 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11851725271040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751432878566216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119360003 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764259612751872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752410992874496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚659361779242496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚739601181543424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119383109 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764259612751872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764262744537088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119383109 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102854238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚5552.178187086 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764262744537088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99261207 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440ÅP^·D2«- _wandbruntime‚5553  _runtime‚5553•k¯ò'¬N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚643020956118016 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11850732191232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218659584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43518372 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11853837598208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218682722 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11830702346968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752542513354240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚751561255463640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218659584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764396350952448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚659477134508544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16456150256128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 $ _timestamp‚1.788433574709832e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218682722 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119404225 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16456095255040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43517856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚659477051373056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚454 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102871596 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119404225 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764393219153920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚739730553116672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99278497 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5553.065247146 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99278497 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102872115 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752542513354240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11850732191232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚739730578544640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119381093 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59353740SÖ¦í­”N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218682722 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11830702346968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752542513354240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚751561255463640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218659584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764396350952448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚659477134508544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16456150256128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 $ _timestamp‚1.788433574709832e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218682722 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119404225 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16456095255040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_splitÖ‘¿o-.small_pool.freed‚43517856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚659477051373056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚454 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102871596 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119404225 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764393219153920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚739730553116672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99278497 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5553.065247146 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99278497 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102872115 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752542513354240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11850732191232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚739730578544640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119381093 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59353740 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚643020956118016 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11850732191232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218659584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43518372 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11853837598208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÆÌÇ›2®- _wandbruntime‚5553  _runtime‚5553ãæWY¯Ó 0loss/FRt3/assignment_loss‚1.9601425186116646 /loss/FRt4/assignment_loss‚4.045667822296555 trainer/global_step‚5549 _runtime‚5553.066569171 2loss/SRqqt3/assignment_loss‚1.1245405170025633 0loss/FBt1/assignment_loss‚0.1433343594955244 .loss/FRt1/detection_loss‚2.237589297037233 1loss/SRqqt4/detection_loss‚2.5890166640281675 /loss/FBt2/detection_loss‚2.4005483809639427 .loss/FRt2/detection_loss‚2.335819074740793 .loss/FRt3/detection_loss‚2.225057360815241 .loss/FRt4/detection_loss‚2.866657943338961 1loss/SRqqt1/detection_loss‚1.4408909490420705 %loss/total_loss‚44.46124979384936 /loss/FRt1/assignment_loss‚2.414002466201782 2loss/SRqqt4/assignment_loss‚1.7626468110829592 /loss/FBt3/detection_loss‚2.2487853041716983 % _timestamp‚1.7884335747113047e+09 1loss/SRqqt1/assignment_loss‚1.370035180792449 1loss/SRqqt2/assignment_loss‚2.125549504602397 1loss/FBt3/assignment_loss‚0.25361542024821965 1loss/SRqqt3/detection_loss‚1.2023745972444029 epoch‚4 1loss/FBt4/assignment_loss‚0.17129261748351413 1loss/SRqqt2/detection_loss‚2.6015401460506298 _step‚455 /loss/FRt2/assignment_loss‚4.153449468123607 1loss/FBt2/assignment_loss‚0.22428154824592597 /loss/FBt1/detection_loss‚0.9704991665092253 /loss/FBt4/detection_loss‚1.5939126757198367ˆ$íÒ^°Ø 1loss/FBt4/assignment_loss‚0.17129261748351413 1loss/SRqqt2/detection_loss‚2.6015401460506298 _step‚455 /loss/FRt2/assignment_loss‚4.153449468123607 1loss/FBt2/assignment_loss‚0.22428154824592597 /loss/FBt1/detection_loss‚0.9704991665092253 /loss/FBt4/detection_loss‚1.5939126757198367 0loss/FRt3/assignment_loss‚1.9601425186116646 /loss/FRt4/assignment_loss‚4.045667822296555 trainer/global_step‚5549 _runtime‚5553.066569171 2loss/SRqqt3/assignment_loss‚1.1245405170025633 0loss/FBt1/assignment_loss‚0.1433343594955244 .loss/FRt1/detection_loss‚2.237589297037233 1loss/SRqqt4/detection_loss‚2.5890166640281675 /loss/FBt2/detection_loss‚2.4005483809639427 .loss/FRt2/detection_loss‚2.335819074740793 .loss/FRt3/detection_loss‚2.225057360815241 .loss/FRt4/detection_loss‚2.866657943338961 1loss/SRqqt1/detection_loss‚1.4408909490420705 %loss/total_loss‚44.46124979384936 /loss/FRt1/assignment_loss‚2.414002466201782 2loss/SRqqt4/assignment_loss‚1.7626468110829592 /loss/FBt3/detection_loss‚2.2487853041716983 % _timestamp‚1.7884335747113047e+09 1loss/SRqqt1/assignment_loss‚1.370035180792449 1loss/SRqqt2/assignment_loss‚2.125549504602397 1loss/FBt3/assignment_loss‚0.25361542024821965 1loss/SRqqt3/detection_loss‚1.2023745972444029 epoch‚4ǦÌK¿±‚(:’ ²©åÔŸ˜–µ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚281.268' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.25359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Uv²‚(:ë ²©åÔè¶ disk.sda3.in‚0.203125 disk.sda3.out‚ 210.76171875 disk.drbd2265.out‚0 disk.dm-0.in‚ 466.734375 disk.sda2.out‚0 disk.rbd0.out‚0 network.sent‚ 1.3378326e+07$ disk./.usageGB‚280.8484802246094! proc.memory.rssMB‚ 2935.421875 disk.drbd2266.in‚0 disk.sda1.in‚0 network.recv‚ 1.93686e+06 disk.sda.out‚ 210.76171875 disk.dm-1.in‚0.203125$ memory_percent‚88.17293643951416* proc.memory.percent‚17.916393280029297 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 466.734375 disk.dm-1.out‚ 210.76171875 disk.drbd2266.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚46134.51025390625 proc.cpu.threads‚17 disk.rbd0.in‚0) disk./.usagePercent‚18.84797715544417 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1937.74609375 disk.drbd1716.out‚0 disk.sda.in‚0.203125& disk.nvme0n1.out‚46134.51025390625 disk.sda2.in‚0# disk.dm-0.out‚46134.51025390625 cpu‚25.256711218276486 disk.rbd1.out‚ 3.00341796875 disk.nvme0c0n1.in‚ 466.734375-áu³‚(:’ Á©åÔˆþˆµ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dªÔxo´‚(:ä Á©åÔÕÞï¸ proc.cpu.threads‚17% disk./.usageGB‚280.89831161499023 disk.rbd1.out‚ 3.06640625 disk.drbd2080.out‚0$ disk.nvme0n1.out‚46217.607421875! proc.memory.rssMB‚ 2935.421875 network.recv‚ 1.943896e+06! disk.dm-0.out‚46217.607421875 disk.sda.out‚ 211.41015625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.734375$ memory_percent‚88.17441463470459 disk.sda2.in‚0* proc.memory.percent‚17.916393280029297 disk.drbd2265.in‚0 network.sent‚ 1.3487055e+07 disk.rbd1.in‚0& disk.nvme0c0n1.out‚46217.607421875 disk.sda1.out‚0 disk.drbd1716.out‚0) disk./.usagePercent‚18.85132138186395 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 466.734375 disk.dm-0.in‚ 466.734375 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.dm-1.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 211.41015625 disk.drbd2266.out‚0 disk.sda3.out‚ 211.41015625) proc.memory.availableMB‚ 1937.50390625 cpu‚25.256138814747363P×Ø žµ‚(:“ ЩåÔÖË©µ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.480399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÿBm¶‚(:â ЩåÔì­Å¶ disk.nvme0n1.in‚ 466.734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.out‚ 211.796875 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.734375 disk.sda3.out‚ 211.796875 disk.drbd1716.in‚0$ memory_percent‚88.17293643951416 proc.cpu.threads‚17 network.sent‚ 1.3489831e+07 disk.rbd0.in‚0( disk.nvme0c0n1.out‚46295.14208984375 disk.sda1.out‚0 disk.dm-1.out‚ 211.796875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda3.in‚0.203125! proc.memory.rssMB‚ 2935.421875 network.recv‚ 1.945686e+06 disk.rbd0.out‚0& disk.nvme0n1.out‚46295.14208984375) disk./.usagePercent‚18.85366615487006# disk.dm-0.out‚46295.14208984375 disk.drbd2080.in‚0* proc.memory.percent‚17.916393280029297 cpu‚25.25556607892486 disk.nvme0c0n1.in‚ 466.734375 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1937.74609375$ disk./.usageGB‚280.9332504272461 disk.sda.in‚0.203125]Rb2·- _wandbruntime‚5599  _runtime‚5599(ùÕ-ǸÁ & lr-AdamW/pg2‚0.0001328351819458145 trainer/global_step‚5599 % _timestamp‚1.7884336206520665e+09 _runtime‚5599.007153679 _step‚456 & lr-AdamW/pg1‚0.0001328351819458145ËÒ¾Ì¹Æ & lr-AdamW/pg1‚0.0001328351819458145 & lr-AdamW/pg2‚0.0001328351819458145 trainer/global_step‚5599 % _timestamp‚1.7884336206520665e+09 _runtime‚5599.007153679 _step‚456Èߦ.2º- _wandbruntime‚5599  _runtime‚5599€Û5 (»ŠP VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758985697284096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220543262 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120440961 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚648529116594688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758985723675648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _step‚457 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43892429 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11954249984000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43891878 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103748015 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 % _timestamp‚1.7884336206528368e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770939947268096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103748569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚665129095910912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770943079053312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770939947268096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100125413 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11937150269000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120417855 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758985697284096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220543262 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120440961 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚648529116594688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758985723675648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _step‚457 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43892429 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11954249984000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43891878 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103748015 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 % _timestamp‚1.7884336206528368e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770939947268096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103748569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚665129095910912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770943079053312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770939947268096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100125413 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11937150269000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120417855 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚458 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚759117191372288 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59866390 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648641527458816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16602859905536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120462077 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5599 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220604774 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648641499324416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚758136558770648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11939259062308 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120438945 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16602914906624 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100142697 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100142691 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11956362297856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚771076685468672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11959467704832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚759117217763840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120462077 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759117191372288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103766238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220604774 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759117217763840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11936155593176 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120438945 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚758139687667748 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103765702 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚746200428605440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚771073553670144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚771076685468672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11956362297856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088ûê«W'¿˜N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚759117191372288 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59866390 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648641527458816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16602859905536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120462077 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5599 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220604774 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648641499324416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚758136558770648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11939259062308 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120438945 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16602914906624 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100142697 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100142691 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11956362297856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚771076685468672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11959467704832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚759117217763840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120462077 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759117191372288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103766238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220604774 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759117217763840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11936155593176 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120438945 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚758139687667748 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103765702 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚746200428605440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚771073553670144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚771076685468672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11956362297856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100142697 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11959467704832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43899845 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884336215335386e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220581636 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚458 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6ÊWŠÕf2À- _wandbruntime‚5599  _runtime‚5599·H­ ^ÁØ 2loss/SRqqt1/assignment_loss‚0.9947658955271161 _runtime‚5599.890483748 /loss/FRt2/detection_loss‚2.2574325168834015 /loss/FBt3/detection_loss‚2.4139913426978246 epoch‚4 2loss/FBt1/assignment_loss‚0.023501705072449936 .loss/FRt3/detection_loss‚2.406517906961116 1loss/SRqqt2/detection_loss‚2.9225933422213015 .loss/FBt2/detection_loss‚3.094683352443907 &loss/total_loss‚45.082265901005066 trainer/global_step‚5599 /loss/FRt3/assignment_loss‚3.303969158367677 /loss/FRt4/assignment_loss‚4.188503689236111 2loss/SRqqt4/assignment_loss‚2.0568941244573304 /loss/FRt1/detection_loss‚1.6328876220128115 /loss/FRt4/detection_loss‚3.0634163927148887 /loss/FBt1/detection_loss‚0.7482073076294157 /loss/FBt4/detection_loss‚1.9156480614581834 % _timestamp‚1.7884336215354939e+09 1loss/SRqqt3/assignment_loss‚1.363040907929341 1loss/FBt2/assignment_loss‚0.24669393134372009 1loss/SRqqt1/detection_loss‚0.8958692723062149 0loss/SRqqt4/detection_loss‚2.287475222891027 _step‚459 2loss/SRqqt2/assignment_loss‚1.6844456312449083 1loss/SRqqt3/detection_loss‚1.7324887337345702 0loss/FBt3/assignment_loss‚0.1387303574252558 1loss/FBt4/assignment_loss‚0.09826542640898285 0loss/FRt1/assignment_loss‚1.8399603314259474 0loss/FRt2/assignment_loss‚3.7722836686115637¦ˆcÂÝ 2loss/FBt1/assignment_loss‚0.023501705072449936 .loss/FRt3/detection_loss‚2.406517906961116 1loss/SRqqt2/detection_loss‚2.9225933422213015 .loss/FBt2/detection_loss‚3.094683352443907 &loss/total_loss‚45.082265901005066 trainer/global_step‚5599 /loss/FRt3/assignment_loss‚3.303969158367677 /loss/FRt4/assignment_loss‚4.188503689236111 2loss/SRqqt4/assignment_loss‚2.0568941244573304 /loss/FRt1/detection_loss‚1.6328876220128115 /loss/FRt4/detection_loss‚3.0634163927148887 /loss/FBt1/detection_loss‚0.7482073076294157 /loss/FBt4/detection_loss‚1.9156480614581834 % _timestamp‚1.7884336215354939e+09 1loss/SRqqt3/assignment_loss‚1.363040907929341 1loss/FBt2/assignment_loss‚0.24669393134372009 1loss/SRqqt1/detection_loss‚0.8958692723062149 0loss/SRqqt4/detection_loss‚2.287475222891027 _step‚459 2loss/SRqqt2/assignment_loss‚1.6844456312449083 1loss/SRqqt3/detection_loss‚1.7324887337345702 0loss/FBt3/assignment_loss‚0.1387303574252558 1loss/FBt4/assignment_loss‚0.09826542640898285 0loss/FRt1/assignment_loss‚1.8399603314259474 0loss/FRt2/assignment_loss‚3.7722836686115637 2loss/SRqqt1/assignment_loss‚0.9947658955271161 _runtime‚5599.890483748 /loss/FRt2/detection_loss‚2.2574325168834015 /loss/FBt3/detection_loss‚2.4139913426978246 epoch‚4Ë]N]“Â(:ˆ ß©åԌגµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚264.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó›)pÄ‚(:å ß©åÔ™Ìé¶ disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚88.17462921142578& proc.memory.availableMB‚ 1937.46875* proc.memory.percent‚17.916393280029297 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.06640625* disk./.usagePercent‚18.855620004371378 disk.nvme0n1.in‚ 466.734375 disk.sda2.out‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2935.421875 network.sent‚ 1.3493999e+07( disk.nvme0c0n1.out‚46369.11083984375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 cpu‚25.255016230146573 proc.cpu.threads‚17& disk.nvme0n1.out‚46369.11083984375# disk.dm-0.out‚46369.11083984375 disk.sda.out‚ 212.6015625 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 466.734375 disk.sda3.out‚ 212.6015625 disk.sda.in‚0.203125 network.recv‚ 1.947997e+06 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 466.734375% disk./.usageGB‚280.96236419677734 disk.dm-1.out‚ 212.6015625 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.in‚0g Ž²Å‚(:’ î©åÔ©Š±µ gpu.0.gpu‚74.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚277.484' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.49679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GbâåcÆ‚(:Ø î©åÔ…¬ó· disk.dm-0.in‚ 466.734375$ memory_percent‚88.17312717437744! proc.memory.rssMB‚ 2935.421875* disk./.usagePercent‚18.857502171763116 disk.drbd1716.in‚0 disk.drbd2080.in‚0$ disk./.usageGB‚280.9904098510742 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 3.06640625* proc.memory.percent‚17.916393280029297 proc.cpu.threads‚17 network.recv‚ 1.954104e+06 disk.sda.out‚ 212.93359375 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.734375 disk.nvme0n1.out‚ 46454.59375 disk.sda1.in‚0 disk.dm-0.out‚ 46454.59375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.out‚ 46454.59375 disk.sda3.out‚ 212.93359375) proc.memory.availableMB‚ 1937.71484375 network.sent‚ 1.3601013e+07 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.254461945312812 disk.rbd0.in‚0 disk.dm-1.out‚ 212.93359375 disk.nvme0c0n1.in‚ 466.734375*ð|žÇ‚(:“ ý©åÔºåèµ gpu.0.gpu‚82.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚128.652' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.730399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Q×dÈ‚(:Ù ý©åÔØŒ¸ proc.cpu.threads‚17 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 cpu‚25.25391894471215 disk.dm-1.out‚ 213.515625 disk.drbd2266.in‚0% disk.nvme0c0n1.out‚46538.89453125 disk.drbd1716.out‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 1937.7265625* proc.memory.percent‚17.916393280029297 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 213.515625 disk.drbd2266.out‚0 disk.sda1.in‚0 network.sent‚ 1.3607453e+07* disk./.usagePercent‚18.860197419083597 disk.rbd0.out‚0! proc.memory.rssMB‚ 2935.421875$ disk./.usageGB‚281.0305709838867 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 3.06640625 disk.nvme0c0n1.in‚ 466.734375 disk.dm-0.in‚ 466.734375 disk.dm-0.out‚46538.89453125 disk.sda3.out‚ 213.515625 disk.sda1.out‚0 network.recv‚ 1.958857e+06# disk.nvme0n1.out‚46538.89453125 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0$ memory_percent‚88.17305564880371Z«´2É- _wandbruntime‚5645  _runtime‚5645¿kåÇÊÁ ' lr-AdamW/pg1‚0.00013142284344973835 ' lr-AdamW/pg2‚0.00013142284344973835 trainer/global_step‚5649 $ _timestamp‚1.788433667341003e+09 _runtime‚ 5645.69602288 _step‚460rygÝÌËÆ ' lr-AdamW/pg1‚0.00013142284344973835 ' lr-AdamW/pg2‚0.00013142284344973835 trainer/global_step‚5649 $ _timestamp‚1.788433667341003e+09 _runtime‚ 5645.69602288 _step‚460ÌÃk2Ì- _wandbruntime‚5645  _runtime‚5645 µ»A^͉P MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222465314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚765560428085248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚461 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚765560401693696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121498813 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777620281784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚752540856237056 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12042603515208 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44271196 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12059880090624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16746243339776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121475707 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777620281784320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚764583485180232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 $ _timestamp‚1.788433667341826e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670895815797760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44270649 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5645.697131532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654149631666688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚461 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚765560401693696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121498813 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777620281784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚752540856237056 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12042603515208 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44271196 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12059880090624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16746243339776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121475707 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777620281784320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚764583485180232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 $ _timestamp‚1.788433667341826e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670895815797760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878ͨ92Ï- _wandbruntime‚5646  _runtime‚5646WWñü'Ð’N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121519929 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚ 5646.59019786 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚5649 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12044712308516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚765691895781888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚752670278666240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12061992404480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚765691895781888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12061992404480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60379040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44278791 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654262042530816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222526826 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16749110760448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚777757019984896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121519929 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104657297 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60379043 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222503688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚765691922173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788433668234405e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121496797 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101006891 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚765691922173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654262070665216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101006897 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101006891 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101006897'.ËÎÑ—N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12044712308516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚765691895781888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚752670278666240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12061992404480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚765691895781888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12061992404480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60379040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44278791 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654262042530816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222526826 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16749110760448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚777757019984896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121519929 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104657297 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60379043 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222503688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚765691922173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788433668234405e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121496797 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101006891 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚765691922173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654262070665216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101006897 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101006891 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101006897 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121519929 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚ 5646.59019786 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚5649 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Îú€Êñ2Ò- _wandbruntime‚5646  _runtime‚5646…çœ{XÓÒ /loss/FRt1/assignment_loss‚2.083985077707391 0loss/FRt3/assignment_loss‚2.4136395797133448 %loss/total_loss‚44.24563233527554 epoch‚4 _runtime‚5646.591229226 0loss/FRt4/assignment_loss‚3.6038821558157603 2loss/SRqqt3/assignment_loss‚1.1439210070421297 .loss/FBt3/detection_loss‚2.818026728100247 2loss/SRqqt1/assignment_loss‚1.2284187427288744 /loss/FRt1/detection_loss‚2.0652114001282476 1loss/SRqqt1/detection_loss‚1.2313085759361817 .loss/FBt2/detection_loss‚2.026395580768585 2loss/FBt1/assignment_loss‚0.059122385546422816 1loss/FBt2/assignment_loss‚0.38364110270384005 1loss/SRqqt3/detection_loss‚1.3062724634508291 .loss/FBt4/detection_loss‚2.115752510334316 trainer/global_step‚5649 _step‚463 % _timestamp‚1.7884336682363203e+09 2loss/SRqqt4/assignment_loss‚1.7938680117328962 /loss/FBt4/assignment_loss‚0.366384838554567 0loss/FRt2/assignment_loss‚3.8140175292889276 0loss/SRqqt2/assignment_loss‚1.71732559800148 .loss/FRt2/detection_loss‚2.585447448823187 .loss/FRt4/detection_loss‚3.000643374522527 /loss/SRqqt2/detection_loss‚2.68154636932456 1loss/SRqqt4/detection_loss‚2.4927686353524527 /loss/FBt1/detection_loss‚0.7799547477201982 1loss/FBt3/assignment_loss‚0.15888052573230502 /loss/FRt3/detection_loss‚2.3752179462462664Žh$*]Ô× 0loss/FRt4/assignment_loss‚3.6038821558157603 2loss/SRqqt3/assignment_loss‚1.1439210070421297 .loss/FBt3/detection_loss‚2.818026728100247 2loss/SRqqt1/assignment_loss‚1.2284187427288744 /loss/FRt1/detection_loss‚2.0652114001282476 1loss/SRqqt1/detection_loss‚1.2313085759361817 .loss/FBt2/detection_loss‚2.026395580768585 2loss/FBt1/assignment_loss‚0.059122385546422816 1loss/FBt2/assignment_loss‚0.38364110270384005 1loss/SRqqt3/detection_loss‚1.3062724634508291 .loss/FBt4/detection_loss‚2.115752510334316 trainer/global_step‚5649 _step‚463 % _timestamp‚1.7884336682363203e+09 2loss/SRqqt4/assignment_loss‚1.7938680117328962 /loss/FBt4/assignment_loss‚0.366384838554567 0loss/FRt2/assignment_loss‚3.8140175292889276 0loss/SRqqt2/assignment_loss‚1.71732559800148 .loss/FRt2/detection_loss‚2.585447448823187 .loss/FRt4/detection_loss‚3.000643374522527 /loss/SRqqt2/detection_loss‚2.68154636932456 1loss/SRqqt4/detection_loss‚2.4927686353524527 /loss/FBt1/detection_loss‚0.7799547477201982 1loss/FBt3/assignment_loss‚0.15888052573230502 /loss/FRt3/detection_loss‚2.3752179462462664 /loss/FRt1/assignment_loss‚2.083985077707391 0loss/FRt3/assignment_loss‚2.4136395797133448 %loss/total_loss‚44.24563233527554 epoch‚4 _runtime‚5646.591229226π̒ã“Õ‚(:ˆ ŒªåԈŸµ gpu.0.gpu‚92.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚304.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.8344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00 º£iÖ‚(:Þ ŒªåÔ®ïÙ¶ disk.drbd1716.in‚0$ disk.nvme0n1.out‚46608.353515625 disk.dm-0.in‚ 466.734375' proc.memory.availableMB‚ 1937.640625 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚88.17358016967773 network.recv‚ 1.962367e+06 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.sda3.in‚0.203125! proc.memory.rssMB‚ 2935.421875 disk.sda.in‚0.203125! disk.dm-0.out‚46608.353515625 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 213.9609375* proc.memory.percent‚17.916393280029297 cpu‚25.25334931979536 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 213.9609375& disk.nvme0c0n1.out‚46608.353515625* disk./.usagePercent‚18.862880122034902$ disk./.usageGB‚281.0705451965332 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 466.734375 network.sent‚ 1.3612726e+07 disk.sda.out‚ 213.9609375 disk.sda1.in‚0à‡çž×‚(:“ ›ªåÔë°™µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚306.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.270599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥/VBmØ‚(:â ›ªåÔÔ¸ª· disk.rbd1.in‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 466.734375& disk.nvme0c0n1.out‚46653.748046875 disk.drbd1716.in‚0 disk.drbd2080.in‚0$ memory_percent‚88.17358016967773$ disk./.usageGB‚281.0803527832031 disk.sda3.out‚ 214.67578125 disk.drbd2080.out‚0! disk.dm-0.out‚46653.748046875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125' proc.memory.availableMB‚ 1937.640625 network.recv‚ 1.968232e+06 disk.dm-0.in‚ 466.734375 disk.drbd2265.out‚0 disk.dm-1.out‚ 214.67578125* proc.memory.percent‚17.916393280029297 network.sent‚ 1.3719515e+07 disk.drbd1716.out‚0 disk.sda.out‚ 214.67578125 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.734375 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0* disk./.usagePercent‚18.863538317405432 disk.sda3.in‚0.203125 disk.sda1.out‚0$ disk.nvme0n1.out‚46653.748046875 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2935.421875 disk.rbd1.out‚ 3.06640625 cpu‚25.252807924897557òΫ˜žÙ‚(:“ ªªåÔžæé´ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï0ë)WÚ‚(:Ì ªªåÔœ¹¡· disk.drbd2265.out‚0$ memory_percent‚88.17391395568848( proc.memory.availableMB‚ 1937.5859375 network.recv‚ 1.972332e+06 disk.rbd0.out‚0 disk.nvme0c0n1.out‚46727.25 disk.dm-0.in‚ 466.734375# proc.memory.rssMB‚ 2935.47265625 cpu‚25.252276148794053 disk.sda1.in‚0 disk.drbd2080.in‚0) proc.memory.percent‚17.91670322418213% disk./.usageGB‚281.10540771484375 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.sda.in‚0.203125 disk.dm-1.out‚ 215.0546875 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.06640625 disk.nvme0c0n1.in‚ 466.734375 disk.sda3.out‚ 215.0546875 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0n1.out‚46727.25 disk.dm-0.out‚46727.25 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda.out‚ 215.0546875 proc.cpu.threads‚17) disk./.usagePercent‚18.86521977489032 disk.nvme0n1.in‚ 466.734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 network.sent‚ 1.3725331e+07©Š‹2Û- _wandbruntime‚5692  _runtime‚5692y!ÉÜà _step‚464 ' lr-AdamW/pg1‚0.00013000443271521178 ' lr-AdamW/pg2‚0.00013000443271521178 trainer/global_step‚5699 % _timestamp‚1.7884337143222258e+09 _runtime‚5692.677244334h–"™ÎÝÈ ' lr-AdamW/pg1‚0.00013000443271521178 ' lr-AdamW/pg2‚0.00013000443271521178 trainer/global_step‚5699 % _timestamp‚1.7884337143222258e+09 _runtime‚5692.677244334 _step‚464Ð5„!T2Þ- _wandbruntime‚5692  _runtime‚56925L²²(ߊP TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44652444 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚5699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚784303748085760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚771158788487240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122556665 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122533559 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101853807 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122556665 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚772135106103296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚759010706297856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105533881 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚676662670587904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12168615590912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚5692.678150925 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12168615590912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12165510197248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚465 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101853813 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚659770203007488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224387366 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚771155659590348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12144953292492 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60881437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44652982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚784300616300544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚784300616300544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105533881 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚676662670587904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12168615590912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚5692.678150925 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12168615590912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12165510197248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚465 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101853813 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚659770203007488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224387366 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚771155659590348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12144953292492 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60881437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44652982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚784300616300544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚784300616300544 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44660646 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122577781 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105552339 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44660130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16895421199872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884337152126548e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105551820 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _step‚466 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101871091 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101871091 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101871097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60891693 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659882613871616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224425740 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772266600191488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚771290294281764 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659882585737216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16895474103808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676778087975424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚784437354501120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101871097 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12170727918080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772266626583040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚759140128727040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12147062085592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚771287165384664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122577781 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122554649 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12150165554724 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224425740 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12167622511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 _runtime‚5693.568160808 trainer/global_step‚5699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60891690 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224448878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759140103299072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0-“,­'ã™N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224425740 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772266600191488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚771290294281764 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659882585737216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16895474103808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676778087975424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚784437354501120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101871097 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12170727918080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772266626583040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚759140128727040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12147062085592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚771287165384664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122577781 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122554649 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12150165554724 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224425740 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12167622511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 _runtime‚5693.568160808 trainer/global_step‚5699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60891690 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224448878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759140103299072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772266626583040 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122554649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚784434222702592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12167622511104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚784437354501120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12170727918080 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44660646 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122577781 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105552339 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44660130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16895421199872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884337152126548e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105551820 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _step‚466 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101871091 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101871091 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101871097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60891693 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659882613871616ÒZØÖP2ä- _wandbruntime‚5693  _runtime‚5693å“PVåÐ 1loss/SRqqt2/assignment_loss‚1.692474993152751 0loss/FBt3/assignment_loss‚0.7612484477262674 0loss/SRqqt2/detection_loss‚2.413321731819047 _runtime‚5693.569240606 _step‚467 /loss/FRt4/assignment_loss‚4.623946065704028 .loss/FBt1/detection_loss‚0.733822998669693 0loss/FRt1/assignment_loss‚1.6713524609804153 0loss/FRt2/assignment_loss‚3.8640693454515365 1loss/FBt1/assignment_loss‚0.03195817063156664 /loss/FBt2/assignment_loss‚0.263668407400308 .loss/FRt2/detection_loss‚2.519796929189137 0loss/SRqqt3/detection_loss‚1.450141363846722 /loss/FBt3/detection_loss‚2.2228481081815867 0loss/SRqqt1/detection_loss‚1.100848505707228 0loss/SRqqt4/detection_loss‚2.418207516541352 .loss/FBt4/detection_loss‚1.906915661834535 1loss/SRqqt4/assignment_loss‚2.208892024852134 0loss/FBt4/assignment_loss‚0.2637177093298912 /loss/FRt3/detection_loss‚2.0260723700317054 0loss/FRt3/assignment_loss‚2.6016689332631917 2loss/SRqqt3/assignment_loss‚1.2343256824107758 /loss/FRt1/detection_loss‚1.5658043010781209 &loss/total_loss‚44.874480944826345 epoch‚4 % _timestamp‚1.7884337152142859e+09 2loss/SRqqt1/assignment_loss‚0.9217263443318948 /loss/FRt4/detection_loss‚3.2690977454185486 /loss/FBt2/detection_loss‚3.1085551272739065 trainer/global_step‚5699¢VÑ”[æÕ /loss/FRt4/detection_loss‚3.2690977454185486 /loss/FBt2/detection_loss‚3.1085551272739065 trainer/global_step‚5699 1loss/SRqqt2/assignment_loss‚1.692474993152751 0loss/FBt3/assignment_loss‚0.7612484477262674 0loss/SRqqt2/detection_loss‚2.413321731819047 _runtime‚5693.569240606 _step‚467 /loss/FRt4/assignment_loss‚4.623946065704028 .loss/FBt1/detection_loss‚0.733822998669693 0loss/FRt1/assignment_loss‚1.6713524609804153 0loss/FRt2/assignment_loss‚3.8640693454515365 1loss/FBt1/assignment_loss‚0.03195817063156664 /loss/FBt2/assignment_loss‚0.263668407400308 .loss/FRt2/detection_loss‚2.519796929189137 0loss/SRqqt3/detection_loss‚1.450141363846722 /loss/FBt3/detection_loss‚2.2228481081815867 0loss/SRqqt1/detection_loss‚1.100848505707228 0loss/SRqqt4/detection_loss‚2.418207516541352 .loss/FBt4/detection_loss‚1.906915661834535 1loss/SRqqt4/assignment_loss‚2.208892024852134 0loss/FBt4/assignment_loss‚0.2637177093298912 /loss/FRt3/detection_loss‚2.0260723700317054 0loss/FRt3/assignment_loss‚2.6016689332631917 2loss/SRqqt3/assignment_loss‚1.2343256824107758 /loss/FRt1/detection_loss‚1.5658043010781209 &loss/total_loss‚44.874480944826345 epoch‚4 % _timestamp‚1.7884337152142859e+09 2loss/SRqqt1/assignment_loss‚0.9217263443318948Ó¦œ’ç‚(:‡ ¹ªåÔ襵 gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚135.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03•tè‚(:é ¹ªåÔ˜‰ì¶ proc.cpu.threads‚17( disk.nvme0c0n1.out‚46782.91748046875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.06640625) proc.memory.percent‚17.91670322418213 network.sent‚ 1.3730591e+07 network.recv‚ 1.975783e+06# disk.dm-0.out‚46782.91748046875 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 215.66796875 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1937.19140625) disk./.usagePercent‚18.86670410657385% disk./.usageGB‚281.12752532958984 disk.sda3.out‚ 215.66796875 disk.nvme0n1.in‚ 466.734375 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2935.47265625 disk.nvme0c0n1.in‚ 466.734375 disk.dm-0.in‚ 466.734375 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 215.66796875 disk.sda2.in‚0 cpu‚25.25179350603288 disk.rbd0.out‚0& disk.nvme0n1.out‚46782.91748046875# memory_percent‚88.1763219833374S³%é‚(:’ ȪåÔëⶵ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.063' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.41259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õýLjê‚(:ß ȪåÔšÝض# proc.memory.rssMB‚ 2935.51953125 disk.rbd1.in‚0& disk.nvme0c0n1.out‚46861.314453125 disk.dm-0.in‚ 466.734375 disk.sda.in‚0.203125 disk.rbd0.in‚0# memory_percent‚88.1763219833374) proc.memory.percent‚17.91698932647705 network.sent‚ 1.3837678e+07 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.734375$ disk.nvme0n1.out‚46861.314453125 disk.nvme0c0n1.in‚ 466.734375! disk.dm-0.out‚46861.314453125 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.out‚ 216.0234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd1716.in‚0$ disk./.usageGB‚281.1578369140625 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 216.0234375 cpu‚25.251224074789974 network.recv‚ 1.98185e+06) disk./.usagePercent‚18.86873834244092 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1937.19140625 disk.dm-1.out‚ 216.0234375 disk.rbd1.out‚ 3.06640625 disk.sda1.in‚0 disk.drbd1716.out‚0Åßå–“ë‚(:ˆ תåÔß›µ gpu.0.gpu‚81.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚186.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ù£Lnì‚(:ã תåÔª‡ù¶% disk./.usageGB‚281.18747329711914 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 216.80078125 disk.sda3.in‚0.203125 network.sent‚ 1.384272e+07* disk./.usagePercent‚18.870727264974434& disk.nvme0c0n1.out‚46926.154296875# proc.memory.rssMB‚ 2935.51953125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.06640625 disk.drbd2080.out‚0 cpu‚25.250697378236897 disk.dm-0.in‚ 466.734375 disk.sda1.out‚0 disk.sda.out‚ 216.80078125 disk.drbd2265.out‚0 disk.sda1.in‚0$ memory_percent‚88.17625045776367 network.recv‚ 1.985393e+06 disk.nvme0n1.in‚ 466.734375 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 216.80078125 disk.nvme0c0n1.in‚ 466.734375! disk.dm-0.out‚46926.154296875' proc.memory.availableMB‚ 1937.203125 proc.cpu.threads‚17 disk.rbd0.in‚0$ disk.nvme0n1.out‚46926.154296875 disk.sda2.out‚0 disk.drbd1716.in‚0) proc.memory.percent‚17.91698932647705=šä2í- _wandbruntime‚5739  _runtime‚57397…xËÉîà ' lr-AdamW/pg2‚0.00012858024533690592 trainer/global_step‚5749 % _timestamp‚1.7884337610857298e+09 _runtime‚5739.440787819 _step‚468 ' lr-AdamW/pg1‚0.00012858024533690592'/[ÎïÈ _runtime‚5739.440787819 _step‚468 ' lr-AdamW/pg1‚0.00012858024533690592 ' lr-AdamW/pg2‚0.00012858024533690592 trainer/global_step‚5749 % _timestamp‚1.7884337610857298e+09Ôþi2ð- _wandbruntime‚5739  _runtime‚5739­ú´kñŠP TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790980950816768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚682429887277568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17039224212480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚778709836904448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚790984082601984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚778709810512896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682429970426368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790980950816768 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102718013 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123614517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45032359 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17039169198080 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚778709810512896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12250406538700 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123614517 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61394090 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonìUãæ¥itor.on_train_batch_start/allocation.all.current‚23112 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚777734091794248 % _timestamp‚1.7884337610866206e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12271140303872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102718013 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚665390746213888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12271140303872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61394087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚469 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778709836904448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123591411 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚777730962897356 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚790984082601984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12253510007624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102718007 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123591411 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106426449 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106425898 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45031811 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚5739.441986501 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12274245697536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚765480581786624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106£ Î(òP YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚777734091794248 % _timestamp‚1.7884337610866206e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12271140303872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102718013 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚665390746213888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12271140303872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61394087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚469 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778709836904448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123591411 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚777730962897356 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚790984082601984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12253510007624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102718007 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123591411 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106426449 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106425898 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45031811 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚5739.441986501 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12274245697536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚765480581786624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790980950816768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚682429887277568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17039224212480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚778709836904448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚790984082601984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚778709810512896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682429970426368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790980950816768 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102718013 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123614517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45032359 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17039169198080 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚778709810512896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12250406538700 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123614517 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61394090ÕE'6ï2ó- _wandbruntime‚5740  _runtime‚5740y„¶ø'ô”N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106443719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚777862468691672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226370930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚778841330992640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61404340 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665503128943616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚778841304601088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12276358024704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682545321899008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102735291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 trainer/global_step‚5749 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚765609953359872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226370930 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123635633 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45039379 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17042164820992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665503157078016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚777865597588772 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12273252617728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚778841304601088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚765609978787840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226347792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚791114557218816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106444252 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682545238763520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚778841330992640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45039909 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17042109819904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12255618800932 % _timestamp‚1.7884337619964268e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12273252617728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5740.352508016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12276358024704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102735297 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123612501 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚470 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123612501 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102735291 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚778841330992640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45039909 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17042109819904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12255618800932 % _timestamp‚1.7884337619964268e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12273252617728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5740.352508016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12276358024704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102735297 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123612501 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚470 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123612501 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102735291 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106443719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 406632960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚777862468691672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226370930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚778841330992640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61404340 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665503128943616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚778841304601088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12276358024704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682545321899008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102735291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 trainer/global_step‚5749 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚765609953359872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_po[ -H ol.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226370930 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123635633 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45039379 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17042164820992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665503157078016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚777865597588772 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12273252617728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚778841304601088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚765609978787840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226347792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚791114557218816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106444252 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682545238763520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797664417118208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚784309395101256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797664417118208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228231470 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797661285332992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45412926 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688197359586816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884338079792068e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12379875804160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688197276438016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107319666 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45412383 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671011261285888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124672369 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚785284514922496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚785284514922496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671011289420288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚785284541314048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124649263 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12379875804160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚784306266204364 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17186070166528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797664417118208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚784309395101256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797664417118208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228231470 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797661285332992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45412926 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688197359586816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884338079792068e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12379875804160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688197276438016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107319666 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45412383 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671011261285888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124672369 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚785284514922496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚785284514922496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671011289420288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚785284541314048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124649263 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12379875804160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚784306266204364 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17186070166528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107336846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61916990 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚785416035402240 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12381988131328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚797798023533568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚785416035402240 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103599491 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12381988131328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12378882724352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124693485 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚5799 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚785416009010688 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228292982 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228269844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45420391 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12357968578008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12378882724352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚784437771998680 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228292982 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107337384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚772079828848640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚772079803420672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚785416009010688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124670353 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688312612097024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984©C_7'‡™N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124670353 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688312612097024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671123700284416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45419856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚474 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61916993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _runtime‚5787.225757995 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124670353 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17188997045248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228269844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103599497 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124693485 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3277969920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107336846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61916990 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚785416035402240 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12381988131328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚797798023533568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚785416035402240 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103599491 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12381988131328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12378882724352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124693485 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚5799 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚785416009010688 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228292982 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228269844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45420391 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12357968578008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12378882724352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚784437771998680 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228292982 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107337384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚772079828848640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚772079803420672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚785416009010688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448Ú󹺠2ˆ- _wandbruntime‚5787  _runtime‚57877¬%%]‰× /loss/FRt1/detection_loss‚1.6012959268982294 /loss/FRt3/detection_loss‚2.1256955779441027 /loss/FBt4/detection_loss‚3.0759178267584906 epoch‚4 /loss/FRt4/assignment_loss‚4.239171617171344 1loss/SRqqt4/assignment_loss‚2.009832561977448 1loss/FBt3/assignment_loss‚0.09183418643078116 0loss/FRt3/assignment_loss‚2.4021897238233816 1loss/SRqqt1/detection_loss‚1.0237289942698935 .loss/FBt3/detection_loss‚1.768099730213483 % _timestamp‚1.7884338088720725e+09 1loss/SRqqt1/assignment_loss‚1.313099721893158 2loss/SRqqt2/assignment_loss‚1.6294605721060822 1loss/FBt2/assignment_loss‚0.30942421158154804 1loss/SRqqt3/detection_loss‚1.7858474937157105 .loss/FRt2/detection_loss‚2.353769232829412 /loss/FRt4/detection_loss‚2.7397045177571915 0loss/SRqqt2/detection_loss‚2.116192514576563 2loss/FBt1/assignment_loss‚0.015195200357681961 1loss/FBt4/assignment_loss‚0.09503875553814901 1loss/SRqqt4/detection_loss‚2.2412683761888936 /loss/FBt1/detection_loss‚0.7256579966822141 /loss/FBt2/detection_loss‚3.5568565974632897 %loss/total_loss‚44.15046865128906 _runtime‚5787.227047048 2loss/SRqqt3/assignment_loss‚1.4484417426352532 trainer/global_step‚5799 _step‚475 /loss/FRt1/assignment_loss‚1.565287169862966 0loss/FRt2/assignment_loss‚3.9174584026137986Xj—bŠÜ 0loss/FRt3/assignment_loss‚2.4021897238233816 1loss/SRqqt1/detection_loss‚1.0237289942698935 .loss/FBt3/detection_loss‚1.768099730213483 % _timestamp‚1.7884338088720725e+09 1loss/SRqqt1/assignment_loss‚1.313099721893158 2loss/SRqqt2/assignment_loss‚1.6294605721060822 1loss/FBt2/assignment_loss‚0.30942421158154804 1loss/SRqqt3/detection_loss‚1.7858474937157105 .loss/FRt2/detection_loss‚2.353769232829412 /loss/FRt4/detection_loss‚2.7397045177571915 0loss/SRqqt2/detection_loss‚2.116192514576563 2loss/FBt1/assignment_loss‚0.015195200357681961 1loss/FBt4/assignment_loss‚0.09503875553814901 1loss/SRqqt4/detection_loss‚2.2412683761888936 /loss/FBt1/detection_loss‚0.7256579966822141 /loss/FBt2/detection_loss‚3.5568565974632897 %loss/total_loss‚44.15046865128906 _runtime‚5787.227047048 2loss/SRqqt3/assignment_loss‚1.4484417426352532 trainer/global_step‚5799 _step‚475 /loss/FRt1/assignment_loss‚1.565287169862966 0loss/FRt2/assignment_loss‚3.9174584026137986 /loss/FRt1/detection_loss‚1.6012959268982294 /loss/FRt3/detection_loss‚2.1256955779441027 /loss/FBt4/detection_loss‚3.0759178267584906 epoch‚4 /loss/FRt4/assignment_loss‚4.239171617171344 1loss/SRqqt4/assignment_loss‚2.009832561977448 1loss/FBt3/assignment_loss‚0.09183418643078116ÛF´’ž‹‚(:“ “«åÔÝРµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚137.901' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.580200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g]ÒæoŒ‚(:ä “«åÔ¨Šø¶ disk.drbd2080.out‚0 disk.sda.in‚0.203125$ disk.nvme0n1.out‚47193.638671875 disk.dm-1.in‚0.203125 disk.nvme0c0n1.in‚ 466.734375 disk.dm-0.in‚ 466.734375 proc.cpu.threads‚17 network.sent‚ 1.3965463e+07 disk.sda.out‚ 218.92578125 disk.rbd1.in‚0! disk.dm-0.out‚47193.638671875 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0.203125 network.recv‚ 2.001621e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2935.57421875* disk./.usagePercent‚18.877826865913985 disk.drbd1716.in‚0 disk.sda3.out‚ 218.92578125 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.06640625$ memory_percent‚88.18025588989258' proc.memory.availableMB‚ 1936.546875 cpu‚25.24858384579742% disk./.usageGB‚281.29326248168945 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚47193.638671875* proc.memory.percent‚17.917323112487793 disk.nvme0n1.in‚ 466.734375 disk.dm-1.out‚ 218.92578125 disk.drbd2266.in‚0žJNד‚(:ˆ ¢«åÔǘ„µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚281.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬J©?rŽ‚(:ç ¢«åÔ¡ÝÛ¶ disk.drbd2266.out‚0& disk.nvme0c0n1.out‚47261.814453125 cpu‚25.248075023433277* disk./.usagePercent‚18.881788070491286 disk.sda3.out‚ 219.38671875 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 3.06640625 network.recv‚ 2.006171e+06 disk.drbd2265.in‚0$ disk.nvme0n1.out‚47261.814453125 disk.sda2.out‚0 disk.dm-1.in‚0.203125$ memory_percent‚88.17722797393799 proc.cpu.threads‚17 disk.dm-0.in‚ 466.734375 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.dm-1.out‚ 219.38671875# proc.memory.rssMB‚ 2935.57421875* proc.memory.percent‚17.917323112487793 network.sent‚ 1.4071217e+07 disk.sda1.out‚0 disk.sda.out‚ 219.38671875 disk.nvme0n1.in‚ 466.734375 disk.drbd2266.in‚0! disk.dm-0.out‚47261.814453125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 466.734375) proc.memory.availableMB‚ 1937.04296875% disk./.usageGB‚281.35228729248047 disk.drbd1716.in‚0 disk.sda2.in‚07¡b·“‚(:ˆ ±«åÔ…€µ gpu.0.gpu‚84.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.9778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö­¿Nm‚(:â ±«åÔ¿Ñз disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 466.734375 disk.sda.out‚ 219.96875$ memory_percent‚88.17722797393799 cpu‚25.247601444859068 disk.drbd2265.in‚0& disk.nvme0n1.out‚47323.87353515625 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚47323.87353515625 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.010403e+06 disk.sda3.out‚ 219.96875# proc.memory.rssMB‚ 2935.57421875) disk./.usagePercent‚18.88009586068996 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125* proc.memory.percent‚17.917323112487793 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0$ disk./.usageGB‚281.3270721435547 disk.rbd1.out‚ 3.06640625 disk.dm-0.in‚ 466.734375 disk.sda3.in‚0.203125 disk.sda1.out‚0 network.sent‚ 1.4076923e+07 disk.nvme0n1.in‚ 466.734375 disk.dm-1.out‚ 219.96875# disk.dm-0.out‚47323.87353515625 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1937.04296875Ð6ÄJ2‘- _wandbruntime‚5834  _runtime‚58340:U È’Â ' lr-AdamW/pg1‚0.00012571572898500795 ' lr-AdamW/pg2‚0.00012571572898500795 trainer/global_step‚5849 $ _timestamp‚1.788433855680951e+09 _runtime‚5834.035970572 _step‚476#«ÓuÍ“Ç _step‚476 ' lr-AdamW/pg1‚0.00012571572898500795 ' lr-AdamW/pg2‚0.00012571572898500795 trainer/global_step‚5849 $ _timestamp‚1.788433855680951e+09 _runtime‚5834.035970572ÜL¯o•2”- _wandbruntime‚5834  _runtime‚5834ê«9-M•ŠP UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104446413 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104446407 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693963989576704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚5849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104446407 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791859219332096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125707115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚676631832626688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12461313031116 _step‚477 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791859219332096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230153522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791859245723648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12482400517120 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125730221 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108209601 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚693963902233600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108209029 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125730221 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804341619849216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62419387 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125707115 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚676631804492288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17332156950016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104446413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45790211 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984¾þæ(–P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚804344751634432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚569 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3277969920 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804344751634432 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12485505910784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12482400517120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45789642 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚778420256480256 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804341619849216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62419387 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125707115 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚676631804492288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17332156950016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104446413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 406632960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45790211 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104446413 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104446407 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693963989576704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚5849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104446407 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791859219332096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125707115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚676631832626688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12461313031116 _step‚477 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791859219332096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230153522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791859245723648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12482400517120 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125730221 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108209601 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚693963902233600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108209029 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125730221 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚804475226251264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62429640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12463421824216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230215034 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230215034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚676744243490816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108226900 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _step‚478 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12487618237952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104463697 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104463691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚778549678909440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45797792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230191896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚791990739811840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45797260 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚5849 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12487618237952 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚791990739811840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17335028627456 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125751337 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104463697 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚791990713420288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12484512830976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚791016204202788 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚791013075305688 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788433856581946e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17335087822848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456ÛØtﺙ˜N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12487618237952 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚791990739811840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17335028627456 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125751337 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104463697 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚791990713420288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12484512830976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚791016204202788 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚791013075305688 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788433856581946e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17335087822848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚804475226251264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62429640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12463421824216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230215034 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230215034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚676744243490816 ?6DeviceStatsMonitor.on_train_batch_eAçÅTdnd/num_device_alloc‚3180 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108226900 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _step‚478 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12487618237952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104463697 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104463691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚778549678909440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45797792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230191896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚791990739811840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45797260 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚5849Þº,º¡2š- _wandbruntime‚5834  _runtime‚5834Ó¯…k\›Ö /loss/FRt3/assignment_loss‚2.861818646391233 1loss/FBt1/assignment_loss‚0.04431021622555991 0loss/FBt2/assignment_loss‚0.3180071352673824 /loss/FBt2/detection_loss‚2.8904899027612476 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚3.771911166435064 .loss/FRt2/detection_loss‚2.621152859094531 1loss/SRqqt3/detection_loss‚1.6553012984494369 _runtime‚5834.938983093 2loss/SRqqt2/assignment_loss‚1.6222138299096016 1loss/FBt4/assignment_loss‚0.07749664491991227 .loss/FRt3/detection_loss‚2.183383370283991 0loss/SRqqt2/detection_loss‚2.616035963258436 1loss/SRqqt4/detection_loss‚2.4641679437721478 /loss/FBt1/detection_loss‚0.9111944921314716 epoch‚4 2loss/SRqqt3/assignment_loss‚1.2104212867716948 1loss/SRqqt1/detection_loss‚1.1163764127936118 /loss/FBt3/detection_loss‚2.5979546254331414 0loss/FRt1/assignment_loss‚1.4105497388278736 0loss/FBt3/assignment_loss‚0.1697142530677163 /loss/FRt1/detection_loss‚1.4866561045979751 /loss/FBt4/detection_loss‚1.9682923768247877 &loss/total_loss‚43.698514400851145 2loss/SRqqt4/assignment_loss‚1.6935603380312814 /loss/FRt4/detection_loss‚3.4486980192801533 0loss/FRt4/assignment_loss‚3.4410012750064625 1loss/SRqqt1/assignment_loss‚1.117806501316432 % _timestamp‚1.7884338565840473e+09 _step‚479ζjåaœÛ trainer/global_step‚5849 /loss/FRt2/assignment_loss‚3.771911166435064 .loss/FRt2/detection_loss‚2.621152859094531 1loss/SRqqt3/detection_loss‚1.6553012984494369 _runtime‚5834.938983093 2loss/SRqqt2/assignment_loss‚1.6222138299096016 1loss/FBt4/assignment_loss‚0.07749664491991227 .loss/FRt3/detection_loss‚2.183383370283991 0loss/SRqqt2/detection_loss‚2.616035963258436 1loss/SRqqt4/detection_loss‚2.4641679437721478 /loss/FBt1/detection_loss‚0.9111944921314716 epoch‚4 2loss/SRqqt3/assignment_loss‚1.2104212867716948 1loss/SRqqt1/detection_loss‚1.1163764127936118 /loss/FBt3/detection_loss‚2.5979546254331414 0loss/FRt1/assignment_loss‚1.4105497388278736 0loss/FBt3/assignment_loss‚0.1697142530677163 /loss/FRt1/detection_loss‚1.4866561045979751 /loss/FBt4/detection_loss‚1.9682923768247877 &loss/total_loss‚43.698514400851145 2loss/SRqqt4/assignment_loss‚1.6935603380312814 /loss/FRt4/detection_loss‚3.4486980192801533 0loss/FRt4/assignment_loss‚3.4410012750064625 1loss/SRqqt1/assignment_loss‚1.117806501316432 % _timestamp‚1.7884338565840473e+09 _step‚479 /loss/FRt3/assignment_loss‚2.861818646391233 1loss/FBt1/assignment_loss‚0.04431021622555991 0loss/FBt2/assignment_loss‚0.3180071352673824 /loss/FBt2/detection_loss‚2.8904899027612476ß|ˆöUž‚(:“ À«åԣᒵ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚295.847' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.169399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýj@Ìuž‚(:ê À«åÔ¯ñà¶( disk.nvme0c0n1.out‚47389.11572265625 disk.sda3.out‚ 220.36328125 disk.drbd2080.in‚0 disk.drbd2080.out‚0& disk.nvme0n1.out‚47389.11572265625 disk.sda2.in‚0 disk.dm-1.out‚ 220.36328125 cpu‚25.247060830003836 proc.cpu.threads‚17 disk.rbd1.out‚ 3.06640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 466.734375 disk.dm-0.in‚ 466.734375$ memory_percent‚88.17784786224365 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125) proc.memory.availableMB‚ 1936.94140625 network.sent‚ 1.40822e+07 disk.sda.out‚ 220.36328125$ disk./.usageGB‚281.4256896972656# disk.dm-0.out‚47389.11572265625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2935.62109375* proc.memory.percent‚17.917609214782715 network.recv‚ 2.013999e+06* disk./.usagePercent‚18.886714167465136$ŽÃì“Ÿ‚(:ˆ Ï«åԚ̅µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚236.493' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.2986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èèz8r ‚(:ç Ï«åÔØ„²· disk.drbd2080.out‚0 disk.rbd1.out‚ 3.06640625 disk.dm-0.in‚ 466.734375 disk.dm-1.out‚ 221.02734375 disk.drbd2266.in‚0 network.sent‚ 1.4189753e+07% disk./.usageGB‚281.45608139038086 disk.sda.out‚ 221.02734375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚47444.4580078125 cpu‚25.246594646432815) disk./.usagePercent‚18.88875377949043 disk.sda1.in‚0% disk.nvme0n1.out‚47444.4580078125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 466.734375# memory_percent‚88.1779432296753 network.recv‚ 2.01999e+06 disk.sda1.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2935.62109375* proc.memory.percent‚17.917609214782715 disk.sda3.in‚0.203125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125" disk.dm-0.out‚47444.4580078125 proc.cpu.threads‚17 disk.sda3.out‚ 221.02734375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚ 466.734375) proc.memory.availableMB‚ 1936.92578125 disk.drbd1716.in‚0s<,ל¡‚(:‘ Þ«åÔÑîä´ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚283.153' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.63060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P:Œp¢‚(:å Þ«åÔµÂÍ· disk.sda3.out‚ 221.49609375 disk.sda2.out‚0# disk.dm-0.out‚47504.82470703125 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 cpu‚25.24609217155736 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 network.sent‚ 1.4193093e+07$ disk./.usageGB‚281.4918327331543 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 network.recv‚ 2.022045e+06 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.734375 disk.nvme0c0n1.in‚ 466.734375 disk.sda3.in‚0.203125* proc.memory.percent‚17.917633056640625 disk.sda.out‚ 221.49609375) proc.memory.availableMB‚ 1936.78515625 proc.memory.rssMB‚2935.625 disk.sda1.in‚0 disk.dm-1.out‚ 221.49609375 disk.dm-0.in‚ 466.734375$ memory_percent‚88.17880153656006 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚47504.82470703125) disk./.usagePercent‚18.89115308210133 disk.drbd2265.out‚0& disk.nvme0n1.out‚47504.82470703125 disk.rbd0.out‚0«ó“£‚(:ˆ í«åÔüñ“µ gpu.0.gpu‚82.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚300.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ÀÈùr¤‚(:ç í«åÔ¦Þº¶) disk./.usagePercent‚18.89353600023004 disk.dm-1.in‚0.203125 disk.nvme0c0n1.in‚ 466.734375 network.sent‚ 1.4198607e+07 disk.rbd0.out‚0 disk.sda3.out‚ 222.17578125% disk./.usageGB‚281.52733993530273 disk.sda.out‚ 222.17578125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚47554.23681640625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 222.17578125( disk.nvme0c0n1.out‚47554.23681640625 disk.drbd2080.out‚0* proc.memory.percent‚17.917633056640625 disk.sda.in‚0.203125 disk.rbd1.out‚ 3.06640625 disk.dm-0.in‚ 466.734375 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1937.03515625 proc.memory.rssMB‚2935.625 disk.rbd0.in‚0 disk.drbd1716.in‚0 cpu‚25.245561878946177 network.recv‚ 2.025895e+06 disk.nvme0n1.in‚ 466.734375& disk.nvme0n1.out‚47554.23681640625 disk.sda2.in‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0$ memory_percent‚88.17727565765381W”~Ž2¥- _wandbruntime‚5881  _runtime‚5881»«†·É¦Ã % _timestamp‚1.7884339027491686e+09 _runtime‚5881.104227473 _step‚480 ' lr-AdamW/pg1‚0.00012427599697233013 ' lr-AdamW/pg2‚0.00012427599697233013 trainer/global_step‚5899Øõã¿Î§È ' lr-AdamW/pg1‚0.00012427599697233013 ' lr-AdamW/pg2‚0.00012427599697233013 trainer/global_step‚5899 % _timestamp‚1.7884339027491686e+09 _runtime‚5881.104227473 _step‚480àÕ`2ð2¨- _wandbruntime‚5881  _runtime‚5881 J(©ŠP ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚699730620842496 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46170328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17478245009408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811025086150656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12591136017408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105310613 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109102368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚699730533499392 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798433950133248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3288455168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 412309504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126788073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126764967 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126788073 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚481 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811021954365440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12588030623744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46169744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17478185800704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 % _timestamp‚1.7884339027499824e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚797460001715272 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784890131969024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚587 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5899 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811021954365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798433923741696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232075574 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798433950133248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105310613 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 _runtime‚5881.105360882 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126764967 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105310607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232098686 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109101781 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚682252347698688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚798433923741696 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105310607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784890106541056 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12591136017408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚811025086150656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62932037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚682252375833088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12569869746248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚797460001715272 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784890131969024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚587 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5899 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811021954365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798433923741696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1697 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232075574 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798433950133248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105310613 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 _runtime‚5881.105360882 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126764967 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105310607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232098686 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109101781 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚682252347698688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚798433923741696 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105310607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784890106541056 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12591136017408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚811025086150656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62932037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚682252375833088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12569869746248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798433950133248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3288455168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 412309504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126788073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456á¡«º2«- _wandbruntime‚5881  _runtime‚5881j^®Ð¦¬”N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12590142937600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 S²dòçt%DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126786057 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105327891 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46177249 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232113948 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚573 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109119539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109120112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3288455168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884339036395457e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798565417829888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105327897 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚811158692566016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12568875070424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12593248344576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12590142937600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 232113948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62942290 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚570 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46177819 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚699845851005952 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126809189 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232137086 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚785019528970240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚682364786697216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 232137086 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126786057 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚797591507509796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5881.995119283 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12593248344576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚682364758562816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 412309504 trainer/global_step‚5899 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798565417829888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105327897 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚811158692566016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12568875070424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12593248344576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12590142937600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 232113948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62942290 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚570 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46177819 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚699845851005952 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126809189 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232137086 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚785019528970240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚682364786697216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 232137086 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126786057 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚797591507509796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5881.995119283 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12593248344576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚682364758562816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 412309504 trainer/global_step‚5899 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232113948 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚573 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109119539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109120112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3288455168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884339036395457e+09 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚817705420666880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46547082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46546512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17623828538368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127845925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127822819 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106174813 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12672219523532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109991772 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocatedÏÕ¹Ðð‚817705420666880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 trainer/global_step‚5949 # _timestamp‚1.78843395013694e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚804032176125388 _runtime‚5928.492242985 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127822819 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687872890905088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚817702288881664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109991199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚817702288881664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12696766124032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791359956601856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12693660730368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚485 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805008654542848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12696766124032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚705496662331904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234020738 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106174813 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791359982029824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233997626 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805008628151296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚705496747577856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687872919039488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127845925 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106174807 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968{~(¼P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805008628151296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚817705420666880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46547082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46546512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17623828538368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127845925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127822819 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106174813 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12672219523532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109991772 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚817705420666880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 trainer/global_step‚5949 # _timestamp‚1.78843395013694e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚804032176125388 _runtime‚5928.492242985 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127822819 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687872890905088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚817702288881664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109991199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚817702288881664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12696766124032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791359956601856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12693660730368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚485 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805008654542848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12696766124032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚705496662331904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234020738 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106174813 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791359982029824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233997626 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805008628151296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚705496747577856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687872919039488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127845925 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106174807 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805008654542848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚804163681919704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚791489353603072 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127867041 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚817839027082240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚805140122239488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚804166810816804 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234059138 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚791489379031040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚486 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110009631 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127843909 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63454940 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17626683474432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234036000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46554688 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127843909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12698878451200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚687985301769216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚817835895283712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 $ _timestamp‚1.788433951023299e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 trainer/global_step‚5949 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110009069 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17626740572672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚805140148631040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚559 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110009069 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17626740572672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚805140148631040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚559 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚804163681919704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚791489353603072 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127867041 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚817839027082240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚805140122239488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚804166810816804 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234059138 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚791489379031040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚486 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110009631 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127843909 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63454940 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17626683474432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234036000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46554688 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127843909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12698878451200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1697 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚687985301769216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚817835895283712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 $ _timestamp‚1.788433951023299e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 trainer/global_step‚5949æ±x„ 2À- _wandbruntime‚5929  _runtime‚5929ttåê]Á× 2loss/SRqqt2/assignment_loss‚1.9526833412237465 $ _timestamp‚1.788433951025505e+09 0loss/SRqqt2/detection_loss‚2.406130949035287 1loss/FBt1/assignment_loss‚0.02437109510300276 0loss/FBt2/assignment_loss‚0.6336322523328515 1loss/FBt3/assignment_loss‚0.17570676931281923 1loss/FBt4/assignment_loss‚0.15775848841585685 .loss/FRt3/detection_loss‚2.094861055320164 /loss/FBt4/detection_loss‚2.0633198767900467 2loss/SRqqt1/assignment_loss‚1.0885958296379872 /loss/FRt1/detection_loss‚1.8125092306379544 /loss/FRt2/detection_loss‚2.2048409461975114 1loss/SRqqt1/detection_loss‚1.1280994802326352 /loss/FBt1/detection_loss‚0.6031077744469779 /loss/FBt2/detection_loss‚2.7364304780960085 /loss/FBt3/detection_loss‚2.4410684406757355 trainer/global_step‚5949 /loss/FRt4/detection_loss‚3.0235114375750225 1loss/SRqqt3/detection_loss‚1.4947243172024924 1loss/SRqqt4/detection_loss‚2.6652249612591485 $loss/total_loss‚43.7216010324179 2loss/SRqqt4/assignment_loss‚1.4988219704140315 _runtime‚5929.380485053 epoch‚4 0loss/FRt2/assignment_loss‚3.5962719708681106 0loss/FRt3/assignment_loss‚3.2713107387974576 _step‚487 0loss/FRt1/assignment_loss‚1.5712629384913688 /loss/FRt4/assignment_loss‚3.863339936733246 2loss/SRqqt3/assignment_loss‚1.2140167536184427ôP²àbÂÜ 2loss/SRqqt2/assignment_loss‚1.9526833412237465 $ _timestamp‚1.788433951025505e+09 0loss/SRqqt2/detection_loss‚2.406130949035287 1loss/FBt1/assignment_loss‚0.02437109510300276 0loss/FBt2/assignment_loss‚0.6336322523328515 1loss/FBt3/assignment_loss‚0.17570676931281923 1loss/FBt4/assignment_loss‚0.15775848841585685 .loss/FRt3/detection_loss‚2.094861055320164 /loss/FBt4/detection_loss‚2.0633198767900467 2loss/SRqqt1/assignment_loss‚1.0885958296379872 /loss/FRt1/detection_loss‚1.8125092306379544 /loss/FRt2/detection_loss‚2.2048409461975114 1loss/SRqqt1/detection_loss‚1.1280994802326352 /loss/FBt1/detection_loss‚0.6031077744469779 /loss/FBt2/detection_loss‚2.7364304780960085 /loss/FBt3/detection_loss‚2.4410684406757355 trainer/global_step‚5949 /loss/FRt4/detection_loss‚3.0235114375750225 1loss/SRqqt3/detection_loss‚1.4947243172024924 1loss/SRqqt4/detection_loss‚2.6652249612591485 $loss/total_loss‚43.7216010324179 2loss/SRqqt4/assignment_loss‚1.4988219704140315 _runtime‚5929.380485053 epoch‚4 0loss/FRt2/assignment_loss‚3.5962719708681106 0loss/FRt3/assignment_loss‚3.2713107387974576 _step‚487 0loss/FRt1/assignment_loss‚1.5712629384913688 /loss/FRt4/assignment_loss‚3.863339936733246 2loss/SRqqt3/assignment_loss‚1.2140167536184427ç÷ùþ[“Â(:ˆ ©¬åÔí—¹µ gpu.0.gpu‚76.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚300.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.1958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ÄÝ*rÄ‚(:ç ©¬åÔÔÛñ·* disk./.usagePercent‚18.897580151248434 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 466.734375 disk.sda1.in‚0 disk.nvme0n1.in‚ 466.734375 proc.cpu.threads‚17 network.sent‚ 1.4324148e+07# disk.dm-0.out‚47772.45751953125 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 224.20703125$ memory_percent‚88.18094730377197 proc.memory.rssMB‚2935.625$ disk./.usageGB‚281.5876007080078 disk.rbd1.out‚ 3.06640625( disk.nvme0c0n1.out‚47772.45751953125 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125) proc.memory.availableMB‚ 1936.43359375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.917633056640625 cpu‚25.243582508654097 network.recv‚ 2.045477e+06 disk.drbd2266.out‚0 disk.dm-1.out‚ 224.20703125 disk.nvme0c0n1.in‚ 466.734375 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 224.20703125& disk.nvme0n1.out‚47772.45751953125ÅTM§žÅ‚(:“ ¸¬åÔ¬Ÿ§µ gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚133.507' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.701400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àPÕvÆ‚(:ë ¸¬åÔ§òÍ· network.sent‚ 1.4430904e+07# disk.dm-0.out‚47819.42236328125 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.sda.out‚ 224.59765625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚47819.42236328125$ memory_percent‚88.17791938781738* proc.memory.percent‚17.917656898498535 network.recv‚ 2.050178e+06 disk.sda3.out‚ 224.59765625 disk.sda1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2935.62890625 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125* disk./.usagePercent‚18.897756540439524$ disk./.usageGB‚281.5902290344238 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 224.59765625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.dm-0.in‚ 466.734375 disk.nvme0n1.in‚ 466.734375 disk.drbd2265.out‚0& disk.nvme0n1.out‚47819.42236328125 disk.nvme0c0n1.in‚ 466.734375 proc.cpu.threads‚17 disk.sda.in‚0.203125 disk.drbd2265.in‚0( proc.memory.availableMB‚ 1936.9296875 cpu‚25.243057530446627ˆWÌÅ“Ç‚(:ˆ ǬåÔÚ‚µ gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚152.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÛsqÈ‚(:æ ǬåÔ¼¤â· disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2935.62890625* proc.memory.percent‚17.917656898498535 proc.cpu.threads‚17 disk.rbd1.out‚ 3.06640625 disk.sda3.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚47877.4501953125 disk.sda3.out‚ 225.23828125 disk.drbd2080.out‚0% disk.nvme0n1.out‚47877.4501953125 disk.dm-1.out‚ 225.23828125 disk.sda.out‚ 225.23828125 disk.rbd0.in‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 1936.6875 network.sent‚ 1.4436153e+07 disk.sda1.in‚0' disk.nvme0c0n1.out‚47877.4501953125 network.recv‚ 2.054554e+06 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.734375 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 466.734375$ memory_percent‚88.17939758300781 cpu‚25.242573243547636* disk./.usagePercent‚18.898627734078545% disk./.usageGB‚281.60321044921875 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.out‚08ï2É- _wandbruntime‚5975  _runtime‚5975ž@„ÉÊà ' lr-AdamW/pg1‚0.00012138308540095692 ' lr-AdamW/pg2‚0.00012138308540095692 trainer/global_step‚5999 % _timestamp‚1.7884339973454258e+09 _runtime‚5975.700508632 _step‚488C!„\ÎËÈ _runtime‚5975.700508632 _step‚488 ' lr-AdamW/pg1‚0.00012138308540095692 ' lr-AdamW/pg2‚0.00012138308540095692 trainer/global_step‚5999 % _timestamp‚1.7884339973454258e+09耡DÑ2Ì- _wandbruntime‚5975  _runtime‚5975iÖL¿ (͇P [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811583332560896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚824382623397888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46926563 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚810607479432396 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12802396230656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693493462245888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46926017 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17769857470976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12799290836992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110883903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12777672769740 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811583332560896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚711263236568064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚797829832090624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811583358952448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107039013 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107039007 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128903777 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12799290836992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128880671 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚824385755183104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12802396230656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12780776238664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811583358952448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63957337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17769802456576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚824385755183104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235919678 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚824382623397888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚797829806662656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚810610608329288 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63957340 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128880671 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693493434111488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107039013 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128903777 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235942790 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚711263319716864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _runtime‚5975.701552233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 " _timestamp‚1.7884339973462e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107039007 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110883354 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5999 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700÷ïÊ€(ÎŒP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128880671 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693493434111488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107039013 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128903777 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235942790 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚711263319716864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _runtime‚5975.701552233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 " _timestamp‚1.7884339973462e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107039007 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110883354 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5999 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811583332560896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚824382623397888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46926563 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚810607479432396 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12802396230656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693493462245888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46926017 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17769857470976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12799290836992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110883903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12777672769740 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811583332560896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚711263236568064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚797829832090624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811583358952448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107039013 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107039007 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128903777 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12799290836992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128880671 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚824385755183104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12802396230656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12780776238664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811583358952448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63957337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17769802456576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚824385755183104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235919678 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚824382623397888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚797829806662656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚810610608329288 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63957340é„Ê}­2Ï- _wandbruntime‚5976  _runtime‚5976Ò eЙДN F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693605844975616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12801403150848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107056291 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235958052 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110901714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128901761 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107056297 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128924893 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63967593 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235981190 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚810738985226712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107056297 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46934121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46933599 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811714826649088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12782885031972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17772729019904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12804508557824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811714853040640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811714826649088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚824516229799936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 trainer/global_step‚5999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128901761 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110901189¸¬Úó'Ñ™N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚824519361598464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚810742114123812 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235981190 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚711378573995520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824519361598464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63967590 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12804508557824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚711378657131008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚797959229091840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107056291 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693605873110016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235958052 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110901714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128901761 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107056297 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128924893 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63967593 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235981190 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚810738985226712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107056297 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46934121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46933599 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811714826649088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12782885031972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17772729019904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12804508557824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811714853040640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811714826649088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚824516229799936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 trainer/global_step‚5999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128901761 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110901189 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693605844975616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12801403150848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107056291 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12921660893184 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚834943761487872 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚17929653415424 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚834946115388928 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 _step‚492 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚834943761487872 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚138 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12923990499840 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 108480774 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115672576 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚6073.884643879 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 238733381 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12923990499840 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚47505651 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚820844406185492 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353901056 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 130234324 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17929769088000 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚834946115388928 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚822022124889088 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112408163 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚64902512 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 130252602 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚720282557299712 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 130252602 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚822022124889088 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚702352921532928 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚702352903884288 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12899804957204 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚12902132648044 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 238715098 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12921660893184 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚834943761487872 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚17929653415424 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚834946115388928 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 _step‚492 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚834943761487872 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚138 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12923990499840 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 108480774 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115672576 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚6073.884643879 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 238733381 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12923990499840 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚47505651 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚820844406185492 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353901056 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 130234324 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17929769088000 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚834946115388928 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚822022124889088 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112408163 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚64902512 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 130252602 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚720282557299712 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 130252602 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚822022124889088 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚702352921532928 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚702352903884288 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12899804957204 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚12902132648044 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 238715098 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 238735990 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 238754273 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚47510502 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚17930267105280 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚17930151432704 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112422680 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 130242018 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 108493977 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚835036206016000 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 238754273 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚64912176 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚822111693204480 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚822111693204480 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 108493972 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚808029550828544 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 108493972 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚835033852114944 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚12902629099692 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12922158910464 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚47510364 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 108493977 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚720358332042240 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1302 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚702428198258176 _step‚493ŠJ¿Úæ‚S NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 130260296 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚12924488517120 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606656 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚820932203259052 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 238735990 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 238754273 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚47510502 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚17930267105280 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚17930151432704 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 2457232998;ƒšL® 4 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112422680 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 130242018 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 108493977 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚835036206016000 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 238754273 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚64912176 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚822111693204480 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚822111693204480 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 108493972 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚808029550828544 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 108493972 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚835033852114944 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚12902629099692 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12922158910464 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚47510364 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 108493977 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚720358332042240 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1302 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚702428198258176 _step‚493 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 130242018 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚835036206016000 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚835033852114944 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚12922158910464 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 130260296 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚820929852237396 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 trainer/global_step‚6044 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112422540 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚64912178 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚702428180609536 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚12900301408852 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 238735990 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚138 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 % _timestamp‚1.7884340965927896e+09 _runtime‚6074.948249254 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚822111717498880 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12924488517120 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 133321216 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021656 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18278 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2353901056 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115672576 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18283 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚822111717498880 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027497198432683945 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5247095227241516 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39212268590927124 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.428507924079895 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4375 particle/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.35309892892837524 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.057139184325933456 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333730697632 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9880632162094116 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5399371385574341 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6755993366241455 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026729561388492584 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23332028090953827 particle/accuracy_6_of_6‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5770832896232605 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 epoch‚4 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39209043979644775 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831715822219849 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5440971851348877 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3720890283584595 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3703787326812744 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/accuracy‚0.8408976197242737 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9805555939674377 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2585586905479431 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2959226369857788 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.38014155626296997 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5454058647155762 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2487669438123703 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12645310163497925 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27167943120002747 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006168720778077841 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14270856976509094 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01065028551965952 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19288046658039093 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.37046757340431213 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20881228148937225 particle/accuracy_3_of_5‚1 6 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.166666641831398 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9854788780212402 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9798904657363892 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.251884400844574 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 'particle/f_score‚0.3473738133907318 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4274103343486786 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4812547266483307 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14339393377304077 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.40337908267974854 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5228174328804016 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3634289503097534 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25741907954216003 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26146039366722107 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5251626968383789 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1822916716337204 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4034916162490845 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013294395990669727 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2876262664794922 *jet/accuracy_2_of_3‚0.2279222011566162 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909184575080872 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 +"Purity/1FRt*SRqqt3FBt/event_purity‚0.25 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2319413125514984 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998187780380249 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17150142788887024 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49079132080078125 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997576117515564 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8221520781517029 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.28125 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.37648287415504456 +jet/accuracy_2_of_2‚0.09177885204553604 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08699306100606918 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7"Purity/0FRt1SRqqt1FBt/event_purity‚0.12424286454916 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16750559210777283 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5675234198570251 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9040834903717041 particle/accuracy_1_of_1‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4898506700992584 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 particle/accuracy_2_of_5‚1 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02153565175831318 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.261981725692749 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚45.12017626742582 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5256721377372742 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48717159032821655 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.026187444105744362 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 +jet/accuracy_3_of_6‚0.09615384787321091 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.29301124811172485 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9958611726760864 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24199122190475464 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013987763784825802 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9905474185943604 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3548445701599121 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18110373616218567 *jet/accuracy_2_of_6‚0.5897436141967773 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3440438508987427 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895937442779541 ,jet/accuracy_3_of_3‚0.019793817773461342 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3950617015361786 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03538155183196068 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21720626950263977 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011040423065423965 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07486415654420853 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19122493267059326 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4165002703666687 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912230968475342 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9923516511917114 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.27135708928108215 7validation_average_jet_accuracy‚0.39781302213668823 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7092098593711853 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.34880951046943665 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8287295699119568 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3016900420188904 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3321637213230133 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964454174041748 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3072916865348816 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08240527659654617 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2817448675632477 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.981029212474823 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2690235376358032 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2653246819972992 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.18439717590808868 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3918810486793518 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.38364511728286743 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_p%—dbß9urity‚0.3333333432674408 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32909998297691345 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26768437027931213 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32375210523605347 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00581051642075181 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_6‚1 jet/accuracy_5_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4083545207977295 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872516989707947 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08116663247346878 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2063492089509964 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4138830304145813 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3323867917060852 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9744970798492432 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36145833134651184 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3490724265575409 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1812455803155899 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444626569748 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9940476417541504 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_4‚0.3318049907684326 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4833333492279053 jet/accuracy_4_of_6‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24343234300613403 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_6_of_6‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2516047954559326 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.30218344926834106 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38602641224861145 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4264557957649231 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46575817465782166 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25102102756500244 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894075989723206 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3899815082550049 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4021317660808563 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.306081086397171 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3785170614719391 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22087740898132324 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9462806582450867 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3444443941116333 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49612072110176086 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 jet/accuracy_7_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22200646996498108 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2250395119190216 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1913580298423767 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25740745663642883 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5682691931724548 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4490741193294525 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22001482546329498 particle/accuracy_1_of_3‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2988017499446869 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36636632680892944 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8985315561294556 particle/accuracy_3_of_3‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3025861978530884 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03339861333370209 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.34892526268959045 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2627898156642914 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16750559210777283 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46929827332496643 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3446841835975647 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24848076701164246 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24253779649734497 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9896023273468018 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.996454119682312 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3566466271877289 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2651515305042267 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.003808243665844202 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3928014934062958 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5250319838523865 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666663646698 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 5 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.99847412109375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.990672767162323 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_4_of_4‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 _runtime‚6089.400455936 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5180681347846985 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3585805594921112 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9809941649436951 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16091515123844147 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3539981245994568 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012433863244950771 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013142459094524384 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2507809102535248 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5013856887817383 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.27953213453292847 particle/accuracy_5_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5291242599487305 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.30996668338775635 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18850575387477875 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03231292590498924 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0758947804570198 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12216367572546005 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5319549441337585 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0IëxFë㥠7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32909998297691345 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26768437027931213 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32375210523605347 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00581051642075181 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_6‚1 jet/accuracy_5_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4083545207977295 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872516989707947 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08116663247346878 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2063492089509964 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4138830304145813 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3323867917060852 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9744970798492432 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36145833134651184 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3490724265575409 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1812455803155899 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444626569748 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9940476417541504 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_4‚0.3318049907684326 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4833333492279053 jet/accuracy_4_of_6‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24343234300613403 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_6_of_6‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2516047954559326 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.30218344926834106 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38602641224861145 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4264557957649231 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46575817465782166 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25102102756500244 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894075989723206 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3899815082550049 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4021317660808563 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.306081086397171 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3785170614719391 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22087740898132324 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9462806582450867 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3444443941116333 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49612072110176086 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 jet/accuracy_7_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22200646996498108 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2250395119190216 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1913580298423767 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25740745663642883 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5682691931724548 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4490741193294525 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22001482546329498 particle/accuracy_1_of_3‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2988017499446869 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36636632680892944 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8985315561294556 particle/accuracy_3_of_3‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3025861978530884 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03339861333370209 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.34892526268959045 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2627898156642914 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16750559210777283 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46929827332496643 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3446841835975647 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24848076701164246 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24253779649734497 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9896023273468018 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.996454119682312 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3566466271877289 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2651515305042267 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.003808243665844202 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3928014934062958 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5250319838523865 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666663646698 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 5 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.99847412109375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.990672767162323 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_4_of_4‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 _runtime‚6089.400455936 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5180681347846985 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3585805594921112 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9809941649436951 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16091515123844147 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3539981245994568 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012433863244950771 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013142459094524384 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2507809102535248 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5013856887817383 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.27953213453292847 particle/accuracy_5_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5291242599487305 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.30996668338775635 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18850575387477875 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03231292590498924 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0758947804570198 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12216367572546005 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5319549441337585 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.272972971200943 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1973462849855423 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.625947117805481 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚6044 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41160428524017334 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.33902832865715027 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 _step‚494 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 $ _timestamp‚1.788434111042993e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922897815704346 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3492063581943512 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.31328296661376953 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22938434779644012 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28493866324424744 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.31437912583351135 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3005056381225586 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.37046757340431213 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.34252873063087463 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011290322057902813 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02216748706996441 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07857143133878708 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.41516441106796265 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19468526542186737 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027497198432683945 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/*FRt*SRqqt1FBt/SRqqt_purã³L7×Lity‚0.5247095227241516 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39212268590927124 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.428507924079895 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4375 particle/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.35309892892837524 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.057139184325933456 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333730697632 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9880632162094116 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5399371385574341 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6755993366241455 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026729561388492584 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23332028090953827 particle/accuracy_6_of_6‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5770832896232605 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 epoch‚4 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39209043979644775 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831715822219849 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5440971851348877 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3720890283584595 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3703787326812744 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/accuracy‚0.8408976197242737 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9805555939674377 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2585586905479431 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2959226369857788 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.38014155626296997 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5454058647155762 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2487669438123703 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12645310163497925 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27167943120002747 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006168720778077841 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14270856976509094 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01065028551965952 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19288046658039093 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.37046757340431213 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20881228148937225 particle/accuracy_3_of_5‚1 6 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.166666641831398 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9854788780212402 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9798904657363892 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.251884400844574 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 'particle/f_score‚0.3473738133907318 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4274103343486786 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4812547266483307 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14339393377304077 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.40337908267974854 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5228174328804016 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3634289503097534 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25741907954216003 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26146039366722107 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5251626968383789 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1822916716337204 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4034916162490845 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013294395990669727 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2876262664794922 *jet/accuracy_2_of_3‚0.2279222011566162 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909184575080872 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 +"Purity/1FRt*SRqqt3FBt/event_purity‚0.25 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2319413125514984 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998187780380249 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17150142788887024 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49079132080078125 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997576117515564 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8221520781517029 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.28125 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.37648287415504456 +jet/accuracy_2_of_2‚0.09177885204553604 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08699306100606918 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7"Purity/0FRt1SRqqt1FBt/event_purity‚0.12424286454916 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16750559210777283 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5675234198570251 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9040834903717041 particle/accuracy_1_of_1‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4898506700992584 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 particle/accuracy_2_of_5‚1 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02153565175831318 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.261981725692749 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚45.12017626742582 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5256721377372742 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48717159032821655 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.026187444105744362 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 +jet/accuracy_3_of_6‚0.09615384787321091 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.29301124811172485 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9958611726760864 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24199122190475464 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013987763784825802 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9905474185943604 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3548445701599121 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18110373616218567 *jet/accuracy_2_of_6‚0.5897436141967773 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3440438508987427 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895937442779541 ,jet/accuracy_3_of_3‚0.019793817773461342 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3950617015361786 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03538155183196068 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21720626950263977 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011040423065423965 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07486415654420853 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19122493267059326 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4165002703666687 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912230968475342 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9923516511917114 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.27135708928108215 7validation_average_jet_accuracy‚0.39781302213668823 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7092098593711853 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.34880951046943665 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8287295699119568 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3016900420188904 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3321637213230133 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964454174041748 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3072916865348816 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08240527659654617 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2817448675632477 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.981029212474823 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2690235376358032 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2653246819972992 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.18439717590808868 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3918810486793518 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.38364511728286743 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204îûD£é‘ì‚(:† ¿­åÔéÕšµ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.049' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4098 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 > …rí‚(:ç ¿­åÔðÓô¶ network.recv‚ 2.085509e+06$ memory_percent‚88.31684589385986 cpu‚25.288836782455185 network.sent‚ 1.4679208e+07) disk./.usagePercent‚18.91125018555426 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.734375 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.in‚0 disk.sda3.out‚ 229.6953125" proc.memory.rssMB‚ 2957.8984375 proc.cpu.threads‚17$ disk./.usageGB‚281.7912940979004& disk.nvme0n1.out‚48415.91845703125 disk.drbd2266.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚48415.94580078125 disk.drbd2265.out‚0 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 229.6953125 disk.rbd1.out‚ 3.21484375 disk.nvme0c0n1.in‚ 466.734375 disk.sda1.in‚0) proc.memory.availableMB‚ 1914.16796875 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 229.6953125* proc.memory.percent‚18.053579330444336 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚48415.90283203125 disk.drbd2080.in‚0e'%žî‚(:“ έåÔÒªù´ gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚145.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.138400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘§Ìqï‚(:æ έåÔÚÜÛ¶ disk.sda2.in‚0' disk.nvme0c0n1.out‚53918.5576171875" disk.dm-0.out‚53918.5576171875 proc.memory.rssMB‚ 2959.40625 disk.dm-1.out‚ 230.13671875 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.21484375( proc.memory.availableMB‚ 4472.0546875 cpu‚25.286500611816646 network.sent‚ 1.4803269e+07 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0* proc.memory.percent‚18.062782287597656 network.recv‚ 2.092491e+06 disk.rbd0.in‚0 disk.rbd1.in‚0% disk./.usageGB‚283.62492752075195 disk.dm-0.in‚ 466.734375 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.dm-1.in‚0.203125$ memory_percent‚72.70474433898926 disk.nvme0c0n1.in‚ 466.734375 disk.nvme0n1.in‚ 466.734375 disk.drbd2266.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚19.034306863082747 disk.sda3.out‚ 230.13671875 disk.drbd1716.out‚0 disk.sda.out‚ 230.13671875 disk.drbd2265.out‚0% disk.nvme0n1.out‚53918.5576171875Sõç‘2ð- _wandbruntime‚6106  _runtime‚6106”Ê .Éñà ' lr-AdamW/pg1‚0.00011993050872069705 ' lr-AdamW/pg2‚0.00011993050872069705 trainer/global_step‚6049 % _timestamp‚1.7884341281390407e+09 _runtime‚6106.494134672 _step‚495™…ÛÎòÈ % _timestamp‚1.7884341281390407e+09 _runtime‚6106.494134672 _step‚495 ' lr-AdamW/pg1‚0.00011993050872069705 ' lr-AdamW/pg2‚0.00011993050872069705 trainer/global_step‚6049ï'‚~Û2ó- _wandbruntime‚6106  _runtime‚6106”$†_Ö$ô‰P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70565888 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚836738680308224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130425182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12936313022976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12939418416640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130448288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130425182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47601808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚809651481539584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚823802367285248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚836741812093440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚836738680308224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108734681 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚703856454701568 trainer/global_step‚6049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823802393676800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42431488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112680620 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17948249468416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚822565901116734 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130448288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚703856426567168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚809651506967552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6507881gg95 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚822569030013626 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚823802393676800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0€yÜä(õŽP [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12936313022976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12939418416640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130448288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130425182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47601808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚809651481539584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚823802367285248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚836741812093440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚836738680308224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108734681 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚703856454701568 trainer/global_step‚6049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823802393676800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42431488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112680620 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17948249468416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚822565901116734 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130448288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚703856426567168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚809651506967552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65078815 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚822569030013626 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚823802393676800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70565888 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚836738680308224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130425182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984ðkµ2ö- _wandbruntime‚6107  _runtime‚6107†ßØ'÷”N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚836875418508800 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚703968837431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65089065 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112698942 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239221375 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12941530743808 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12941530743808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130469404 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836872286710272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108751971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚823933887764992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚822697406911050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚703968865565696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42418176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6107.388041348 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239221375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65089068 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884341290321898e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚822700535808150 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823933887764992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130469404 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721920181512192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112698502 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47609437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108751965 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚809780878540800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239198237 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108751971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721920110959616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚809780903968768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12919631839382 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚823933861373440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12938425336832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130446272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚836875418508800 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032Ú¤Õœ'ø™N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130469404 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721920181512192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112698502 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47609437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108751965 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚809780878540800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239198237 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108751971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721920110959616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚809780903968768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12919631839382 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚823933861373440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12938425336832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130446272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚836875418508800 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚836875418508800 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚703968837431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65089065 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112698942 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239221375 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12941530743808 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12941530743808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130469404 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836872286710272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108751971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚823933887764992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚822697406911050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚703968865565696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42418176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6107.388041348 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239221375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65089068 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884341290321898e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚822700535808150 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823933887764992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302ñŽ¼8­2ù- _wandbruntime‚6107  _runtime‚6107cÄŽ\úÖ 1loss/SRqqt3/detection_loss‚1.7362717430178936 .loss/FRt2/detection_loss‚2.420100533685019 /loss/FBt2/detection_loss‚2.5064510464668275 0loss/FBt2/assignment_loss‚0.4823392548931603 /loss/FBt4/detection_loss‚1.7244971754650276 trainer/global_step‚6049 /loss/FRt1/detection_loss‚1.6374535300933264 1loss/SRqqt2/detection_loss‚2.4355171129984012 /loss/FBt1/detection_loss‚0.8073855663960179 % _timestamp‚1.7884341290342376e+09 _step‚498 0loss/FRt1/assignment_loss‚1.4233374174903421 2loss/SRqqt3/assignment_loss‚1.6285851397694877 0loss/FBt4/assignment_loss‚0.1277846659160673 /loss/FRt3/detection_loss‚2.7097745690590296 .loss/FBt3/detection_loss‚2.435916832902215 epoch‚5 /loss/FRt3/assignment_loss‚3.241958576899308 _runtime‚ 6107.38925003 %loss/total_loss‚44.12312248380533 1loss/FBt3/assignment_loss‚0.25412328137526236 /loss/FRt4/detection_loss‚2.9071075217798352 1loss/SRqqt1/detection_loss‚0.8860530752565835 1loss/SRqqt4/detection_loss‚2.5803253786130385 /loss/FRt2/assignment_loss‚3.516734533531721 /loss/FRt4/assignment_loss‚4.147484872490168 2loss/SRqqt1/assignment_loss‚1.1443725606058162 2loss/SRqqt2/assignment_loss‚1.2834011505850975 2loss/SRqqt4/assignment_loss‚2.0759430652992292 2loss/FBt1/assignment_loss‚0.010203879216460147^8 8‚ûÛ %loss/total_loss‚44.12312248380533 1loss/FBt3/assignment_loss‚0.25412328137526236 /loss/FRt4/detection_loss‚2.9ÿbó¯ß071075217798352 1loss/SRqqt1/detection_loss‚0.8860530752565835 1loss/SRqqt4/detection_loss‚2.5803253786130385 /loss/FRt2/assignment_loss‚3.516734533531721 /loss/FRt4/assignment_loss‚4.147484872490168 2loss/SRqqt1/assignment_loss‚1.1443725606058162 2loss/SRqqt2/assignment_loss‚1.2834011505850975 2loss/SRqqt4/assignment_loss‚2.0759430652992292 2loss/FBt1/assignment_loss‚0.010203879216460147 1loss/SRqqt3/detection_loss‚1.7362717430178936 .loss/FRt2/detection_loss‚2.420100533685019 /loss/FBt2/detection_loss‚2.5064510464668275 0loss/FBt2/assignment_loss‚0.4823392548931603 /loss/FBt4/detection_loss‚1.7244971754650276 trainer/global_step‚6049 /loss/FRt1/detection_loss‚1.6374535300933264 1loss/SRqqt2/detection_loss‚2.4355171129984012 /loss/FBt1/detection_loss‚0.8073855663960179 % _timestamp‚1.7884341290342376e+09 _step‚498 0loss/FRt1/assignment_loss‚1.4233374174903421 2loss/SRqqt3/assignment_loss‚1.6285851397694877 0loss/FBt4/assignment_loss‚0.1277846659160673 /loss/FRt3/detection_loss‚2.7097745690590296 .loss/FBt3/detection_loss‚2.435916832902215 epoch‚5 /loss/FRt3/assignment_loss‚3.241958576899308 _runtime‚ 6107.38925003ò*ðé‚žü‚(:“ Ý­åÔ縥µ gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚42.0 gpu.0.powerWatts‚203.515' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.702999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨/]3uý‚(:ê Ý­åÔʵ· disk.rbd1.in‚0 disk.rbd1.out‚ 3.2470703125 disk.sda.in‚0.203125 disk.rbd0.in‚0 network.recv‚ 2.096694e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0$ disk./.usageGB‚283.5698356628418 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 230.87890625 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 cpu‚25.285919870414997 network.sent‚ 1.4808746e+07 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.734375$ memory_percent‚72.70514965057373 proc.cpu.threads‚17 disk.sda3.out‚ 230.87890625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0) disk./.usagePercent‚19.03060960227325 disk.dm-0.in‚ 466.734375 disk.drbd2265.out‚0* proc.memory.percent‚18.062782287597656 disk.nvme0c0n1.in‚ 466.734375 disk.sda.out‚ 230.87890625 disk.sda2.out‚0) proc.memory.availableMB‚ 4471.98828125 proc.memory.rssMB‚ 2959.40625 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚54949.52783203125# disk.dm-0.out‚54949.52783203125& disk.nvme0n1.out‚54949.52783203125 disk.sda2.in‚0„GZžþ‚(:“ ì­åÔù¶‚µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚303.817' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.763400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘;%rÿ‚(:ç ì­åÔ†ø· disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125) proc.memory.availableMB‚ 4471.94140625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.2470703125 cpu‚25.285342174686306$ disk./.usageGB‚283.6062812805176# disk.dm-0.out‚55012.18408203125 disk.sda1.in‚0 disk.sda.out‚ 231.3359375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚55012.18408203125! proc.memory.rssMB‚ 2959.453125 network.sent‚ 1.4916831e+07 network.recv‚ 2.10328e+06 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 231.3359375 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚0.203125$ memory_percent‚72.70543575286865* proc.memory.percent‚18.063068389892578 disk.nvme0n1.in‚ 466.734375 disk.dm-1.out‚ 231.3359375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚55012.18408203125 disk.dm-0.in‚ 466.734375) disk./.usagePercent‚19.03305549825538 disk.dm-1.in‚0.203125 disk.nvme0c0n1.in‚ 466.734375‡'‚(:“ û­åÔ†ý´ gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚296.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.217600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fj׌n‚(:ã û­åÔææÆ· disk.sda3.out‚ 231.91015625 disk.drbd2080.out‚0 disk.drbd2266.in‚0 network.recv‚ 2.107725e+06 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda.out‚ 231.91015625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.31005859375% disk.nvme0c0n1.out‚55076.41015625 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2959.453125$ disk./.usageGB‚281.7729949951172 disk.nvme0c0n1.in‚ 466.734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 231.91015625* proc.memory.percent‚18.063068389892578 disk.drbd1716.in‚0# disk.nvme0n1.out‚55076.41015625) proc.memory.availableMB‚ 4471.94140625 disk.dm-0.out‚55076.41015625 disk.dm-1.in‚0.203125 network.sent‚ 1.4922867e+07) disk./.usagePercent‚18.91002211741251 disk.sda3.in‚0.203125 disk.drbd2265.in‚0 disk.rbd1.in‚0$ memory_percent‚72.70543575286865 cpu‚25.284770285995133 proc.cpu.threads‚17 disk.dm-0.in‚ 466.734375VzM2‚- _wandbruntime‚6154  _runtime‚6154ç•ÈɃà ' lr-AdamW/pg1‚0.00011847425478731919 ' lr-AdamW/pg2‚0.00011847425478731919 trainer/global_step‚6099 % _timestamp‚1.7884341756727283e+09 _runtime‚6154.027781151 _step‚499ù‚g¡Î„È ' lr-AdamW/pg1‚0.00011847425478731919 ' lr-AdamW/pg2‚0.00011847425478731919 trainer/global_step‚6099 % _timestamp‚1.7884341756727283e+09 _runtime‚6154.027781151 _step‚499óê—W2…- _wandbruntime‚6154  _runtime‚6154.Æ=Ö(†‰P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131483034 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109598887 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47985212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6154.029159193 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113576674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47985730 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109598887 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚727574295993856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚6099 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131483034 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109598881 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚843419014824448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18097247265792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113577195 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816121331600384 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚830377098086400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13045048523264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚843422146609664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109598881 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131506140 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚727574217039360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚830377071694848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚709476997907968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚843422146609664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241105027 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13041943129600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131506140 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788434175673755e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚830377071694848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241081915 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚830377098086400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13045048523264 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788434175673755e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚830377071694848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241081915 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚830377098086400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13045048523264 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚843419014824448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18097247265792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113577195 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816121331600384 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚830377098086400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13045048523264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚843422146609664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109598881 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131506140 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚727574217039360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚830377071694848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚709476997907968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ôhc\)2ˆ- _wandbruntime‚6154  _runtime‚6154”{ê9‰”N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13047160850432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131527256 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚829272710218058 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13044055443456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241120289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109616165 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18100193123840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚830508592174592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13025085085590 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131504124 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109616171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816250728601600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241143427 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131527256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚709589380637696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113594364 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47992649 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚843555753025024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚501 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109616171 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241120289 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚843555753025024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚727689652702720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816250754029568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13021981616458 trainer/global_step‚6099(§­B'Š™N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109616165 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18100193123840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚830508592174592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13025085085590 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131504124 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109616171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816250728601600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241143427 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131527256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚709589380637696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113594364 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47992649 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚843555753025024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚501 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109616171 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241120289 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚843555753025024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚727689652702720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816250754029568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13021981616458 trainer/global_step‚6099 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13047160850432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131527256 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚829272710218058 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13044055443456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241120289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700õÅc2‹- _wandbruntime‚6154  _runtime‚6154t=‡Ž[ŒÕ /loss/FRt2/detection_loss‚2.4129928090633492 .loss/FRt4/detection_loss‚3.082845072746277 1loss/SRqqt1/detection_loss‚0.9535901407994654 1loss/FBt1/assignment_loss‚0.01751583548631033 0loss/FBt4/assignment_loss‚0.3013239126099506 1loss/SRqqt2/detection_loss‚2.4243780862201345 /loss/FBt3/detection_loss‚3.5902518298890858 /loss/FRt4/assignment_loss‚4.445218148231507 1loss/FBt2/assignment_loss‚0.23105800972982454 /loss/FBt2/detection_loss‚2.1693012250794306 trainer/global_step‚6099 _step‚502 /loss/FRt2/assignment_loss‚3.247865686049828 .loss/FRt3/detection_loss‚2.601074515283108 epoch‚5 % _timestamp‚1.7884341765778875e+09 /loss/FRt1/assignment_loss‚1.454638749001974 1loss/FBt3/assignment_loss‚0.20080548152313485 _runtime‚ 6154.93282605 2loss/SRqqt1/assignment_loss‚0.9929943972257087 2loss/SRqqt3/assignment_loss‚1.2882028017995943 1loss/SRqqt3/detection_loss‚1.8230445909089055 2loss/SRqqt4/assignment_loss‚2.0169385029003024 .loss/FRt1/detection_loss‚1.435630073389375 1loss/SRqqt4/detection_loss‚2.2741935463512646 /loss/FBt1/detection_loss‚0.9053081949320385 /loss/FBt4/detection_loss‚1.7566878646612167 &loss/total_loss‚44.679429909687535 2loss/SRqqt2/assignment_loss‚2.0379295213655992 /loss/FRt3/assignment_loss‚3.015640914440155\y¹`Ú /loss/FRt3/assignment_loss‚3.015640914440155 /loss/FRt2/detection_loss‚2.4129928090633492 .loss/FRt4/detection_loss‚3.082845072746277 1loss/SRqqt1/detection_loss‚0.9535901407994654 1loss/FBt1/assignment_loss‚0.01751583548631033 0loss/FBt4/assignment_loss‚0.3013239126099506 1loss/SRqqt2/detection_loss‚2.4243780862201345 /loss/FBt3/detection_loss‚3.5902518298890858 /loss/FRt4/assignment_loss‚4.445218148231507 1loss/FBt2/assignment_loss‚0.23105800972982454 /loss/FBt2/detection_loss‚2.1693012250794306 trainer/global_step‚6099 _step‚502 /loss/FRt2/assignment_loss‚3.247865686049828 .loss/FRt3/detection_loss‚2.601074515283108 epoch‚5 % _timestamp‚1.7884341765778875e+09 /loss/FRt1/assignment_loss‚1.454638749001974 1loss/FBt3/assignment_loss‚0.20080548152313485 _runtime‚ 6154.93282605 2loss/SRqqt1/assignment_loss‚0.9929943972257087 2loss/SRqqt3/assignment_loss‚1.2882028017995943 1loss/SRqqt3/detection_loss‚1.8230445909089055 2loss/SRqqt4/assignment_loss‚2.0169385029003024 .loss/FRt1/detection_loss‚1.435630073389375 1loss/SRqqt4/detection_loss‚2.2741935463512646 /loss/FBt1/detection_loss‚0.9053081949320385 /loss/FBt4/detection_loss‚1.7566878646612167 &loss/total_loss‚44.679429909687535 2loss/SRqqt2/assignment_loss‚2.0379295213655992ö‡¬TH’Ž‚(:‡ Š®åÔ×¢¦¶ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.897 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-dFYg‚(:Ü Š®åÔù­þ¶ disk.dm-1.in‚0.203125% disk./.usageGB‚281.80906677246094 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0# disk.nvme0c0n1.out‚ 55134.296875 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 466.734375 disk.sda2.in‚0! proc.memory.rssMB‚ 2959.453125 disk.sda3.out‚ 232.56640625* proc.memory.percent‚18.063068389892578! disk.nvme0n1.out‚ 55134.296875 network.sent‚ 1.4926735e+07* disk./.usagePercent‚18.912442924656286 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 232.56640625& proc.memory.availableMB‚ 4471.90625 disk.rbd1.out‚ 3.31005859375 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.sda2.out‚0$ memory_percent‚72.70565032958984 proc.cpu.threads‚17 network.recv‚ 2.109551e+06 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 466.734375 disk.dm-0.in‚ 466.734375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 cpu‚25.284206165429644 disk.dm-0.out‚ 55134.296875 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.sda.out‚ 232.56640625k´|¡’‚(:‡ ™®åÔ à‡µ gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.897 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iísj‘‚(:ß ™®åÔ³ìé·* disk./.usagePercent‚18.913775699879544* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 network.recv‚ 2.115651e+06 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚55220.330078125$ disk./.usageGB‚281.8289260864258 disk.drbd1716.out‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚55220.330078125 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 466.734375 disk.nvme0c0n1.in‚ 466.734375 disk.sda3.out‚ 233.2890625 disk.sda1.out‚0$ memory_percent‚72.70565032958984 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.373046875! disk.dm-0.out‚55220.330078125 cpu‚25.283673306117976& proc.memory.availableMB‚ 4471.90625 disk.dm-0.in‚ 466.734375 network.sent‚ 1.5034069e+07 disk.drbd2080.in‚0 disk.sda.out‚ 233.2890625 disk.dm-1.out‚ 233.2890625 disk.sda3.in‚0.203125! proc.memory.rssMB‚ 2959.453125 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda1.in‚0Ó¤Áž’‚(:“ ¨®åÔÌ¡ÿ´ gpu.0.gpu‚46.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.462199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰˜¡u“‚(:ê ¨®åÔã’· disk.sda3.in‚0.203125 disk.drbd1716.in‚0 network.recv‚ 2.11696e+06% disk./.usageGB‚281.85130310058594 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.73828125$ disk.nvme0n1.out‚55284.572265625 disk.dm-1.out‚ 233.66015625& disk.nvme0c0n1.out‚55284.572265625 proc.cpu.threads‚17 network.sent‚ 1.5036829e+07 disk.rbd0.in‚0! disk.dm-0.out‚55284.572265625 disk.drbd2080.out‚0 disk.sda.out‚ 233.66015625$ memory_percent‚72.70629405975342) proc.memory.availableMB‚ 4471.80078125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125! proc.memory.rssMB‚ 2959.453125 cpu‚25.28307657775078* disk./.usagePercent‚18.915277440075396 disk.sda3.out‚ 233.66015625* proc.memory.percent‚18.063068389892578 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.73828125 disk.sda2.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0m }Ð2”- _wandbruntime‚6201  _runtime‚6201–½‡ºÉ•Ã ' lr-AdamW/pg1‚0.00011701462708196037 ' lr-AdamW/pg2‚0.00011701462708196037 trainer/global_step‚6149 % _timestamp‚1.7884342231699166e+09 _runtime‚6201.524965308 _step‚503wdù£Î–È ' lr-AdamW/pg1‚0.00011701462708196037 ' lr-AdamW/pg2‚0.00011701462708196037 trainer/global_step‚6149 % _timestamp‚1.7884342231699166e+09 _runtime‚6201.524965308 _step‚503÷Wäš2—- _wandbruntime‚6201  _runtime‚6201hYâÊ(˜‰P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715097512979968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835716507730750 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚733342264643072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13125326069566 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚836951776104448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18244804580352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114471455 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18244751663104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13150678629888 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132563992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚822591181661184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850102481125888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110463081 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132540886 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850099349340672 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13128429538490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835719636627642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132540886 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110463081 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48367343 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚836951802496000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13150678629888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13147573236224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚733342345694720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 $ _timestamp‚1.788434223170678e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚836951802496000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66104112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48367875 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚850099349340672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66104115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715097541114368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243027079 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚6201.525994801 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850102481125888°1=‡Š™ŽP trainer/global_step‚6149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110463087 [HDeviceStatsMoniÂ} XŠ'tor.on_train_batch_start/allocated_bytes.small_pool.freed‚13147573236224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110463087 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66104112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48367875 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚850099349340672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66104115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715097541114368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243027079 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚6201.525994801 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850102481125888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715097512979968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835716507730750 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚733342264643072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13125326069566 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚836951776104448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18244804580352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114471455 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18244751663104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13150678629888 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132563992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚822591181661184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850102481125888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110463081 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132540886 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850099349340672 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13128429538490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835719636627642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132540886 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110463081 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48367343 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚836951802496000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13150678629888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13147573236224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚733342345694720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 $ _timestamp‚1.788434223170678e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚836951802496000 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13152790957056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚715209923844096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850236087541248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114489715 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243065479 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132561976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110480365 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚ 6202.41444366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 243042341 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚835851142422166 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 243065479 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚837083270192640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13152790957056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13130538331798 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114489207 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13127434862666 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 243042341 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110480365 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302ƒgá['œ˜N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850236087541248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114489715 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243065479 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132561976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110480365 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚ 6202.41444366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 243042341 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚835851142422166 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 243065479 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚837083270192640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13152790957056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13130538331798 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114489207 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13127434862666 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 243042341 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110480365 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66114365 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 % _timestamp‚1.7884342240583954e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13152790957056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚715209923844096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249ù~bÈ2- _wandbruntime‚6202  _runtime‚6202ÅŒ3YžÓ 0loss/FBt2/assignment_loss‚0.3095742860962425 .loss/FRt4/detection_loss‚3.437776197086681 epoch‚5 2loss/SRqqt2/assignment_loss‚1.4311468324065209 1loss/FBt4/assignment_loss‚0.13397856664960273 1loss/SRqqt1/detection_loss‚0.9392999843420351 .loss/FRt2/detection_loss‚2.322433328905771 /loss/FRt1/assignment_loss‚1.272465013322376 2loss/SRqqt1/assignment_loss‚0.9797015721526217 1loss/SRqqt3/assignment_loss‚1.181117664639126 2loss/SRqqt4/assignment_loss‚1.3510842825552183 1loss/FBt1/assignment_loss‚0.09795321284453343 /loss/FRt3/detection_loss‚2.3404855863614515 _runtime‚6202.415708517 /loss/FRt2/assignment_loss‚3.690394961556723 /loss/FRt3/assignment_loss‚2.429636764255437 /loss/FRt4/assignment_loss‚4.513652606443926 1loss/SRqqt4/detection_loss‚2.6466875672340393 /loss/FBt1/detection_loss‚0.5865734198975355 /loss/FBt3/detection_loss‚2.7983873983224234 % _timestamp‚1.7884342240606995e+09 1loss/FBt3/assignment_loss‚0.14334253277047537 /loss/SRqqt2/detection_loss‚2.46827530503273 /loss/FBt2/detection_loss‚2.3565479191866787 /loss/FRt1/detection_loss‚1.5309400457652316 1loss/SRqqt3/detection_loss‚1.8048706376085095 /loss/FBt4/detection_loss‚2.1814586706459522 %loss/total_loss‚42.94778435608184 trainer/global_step‚6149 _step‚506¹ÏÊÉQŸØ /loss/FRt2/assignment_loss‚3.690394961556723 /loss/FRt3/assignment_loss‚2.429636764255437 /loss/FRt4/assignment_loss‚4.513652606443926 1loss/SRqqt4/detection_loss‚2.6466875672340393 /loss/FBt1/detection_loss‚0.5865734198975355 /loss/FBt3/detection_loss‚2.7983873983224234 % _timestamp‚1.7884342240606995e+09 1loss/FBt3/assignment_loss‚0.14334253277047537 /loss/SRqqt2/detection_loss‚2.46827530503273 /loss/FBt2/detection_loss‚2.3565479191866787 /loss/FRt1/detection_loss‚1.5309400457652316 1loss/SRqqt3/detection_loss‚1.8048706376085095 /loss/FBt4/detection_loss‚2.1814586706459522 %loss/total_loss‚42.94778435608184 trainer/global_step‚6149 _step‚506 0loss/FBt2/assignment_loss‚0.3095742860962425 .loss/FRt4/detection_loss‚3.437776197086681 epoch‚5 2loss/SRqqt2/assignment_loss‚1.4311468324065209 1loss/FBt4/assignment_loss‚0.13397856664960273 1loss/SRqqt1/detection_loss‚0.9392999843420351 .loss/FRt2/detection_loss‚2.322433328905771 /loss/FRt1/assignment_loss‚1.272465013322376 2loss/SRqqt1/assij}{á gnment_loss‚0.9797015721526217 1loss/SRqqt3/assignment_loss‚1.181117664639126 2loss/SRqqt4/assignment_loss‚1.3510842825552183 1loss/FBt1/assignment_loss‚0.09795321284453343 /loss/FRt3/detection_loss‚2.3404855863614515 _runtime‚6202.415708517útïk‚“ ‚(:ˆ ·®åÔ«‚‰µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚297.097' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F(™Ws¡‚(:è ·®åÔðí¡· disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.out‚55353.42578125 cpu‚25.282513769227144 disk.sda1.in‚0 disk.sda.out‚ 234.47265625 disk.nvme0n1.in‚ 466.73828125 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.73828125$ memory_percent‚72.71053791046143* disk./.usagePercent‚18.916235676276056 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.dm-1.out‚ 234.47265625 disk.drbd2266.in‚0 network.recv‚ 2.120086e+06$ disk./.usageGB‚281.8655815124512 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚55353.42578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 234.47265625) proc.memory.availableMB‚ 4471.10546875% disk.nvme0c0n1.out‚55353.42578125 disk.sda3.in‚0.203125! proc.memory.rssMB‚ 2959.453125 network.sent‚ 1.5041928e+07* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 disk.rbd1.out‚ 3.373046875 disk.drbd2080.in‚0JÍR÷¢‚(:’ Æ®åÔ™ÏÝ´ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.732' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.54640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ógmo£‚(:ä Æ®åÔ–Ω· disk.rbd1.out‚ 3.373046875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2959.453125 cpu‚25.28199520101271 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.73828125& disk.nvme0c0n1.out‚55413.935546875 disk.sda.in‚0.203125 disk.rbd0.out‚0 network.recv‚ 2.124617e+06 disk.dm-0.in‚ 466.73828125 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.063068389892578 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0$ memory_percent‚72.70874977111816 disk.sda2.in‚0 disk.sda.out‚ 234.9765625 proc.cpu.threads‚17 network.sent‚ 1.5150245e+07 disk.dm-1.out‚ 234.9765625$ disk.nvme0n1.out‚55413.935546875$ disk./.usageGB‚281.8825874328613 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 4471.3984375( disk./.usagePercent‚18.9173769578637 disk.dm-1.in‚0.203125 disk.rbd1.in‚0! disk.dm-0.out‚55413.935546875 disk.sda3.out‚ 234.9765625 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.73828125ò¡T “¤‚(:ˆ Õ®åÔöድ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÚÓÆz¥‚(:ï Õ®åÔØÍß· network.recv‚ 2.127822e+06& disk.nvme0n1.out‚55482.04052734375 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.73828125$ memory_percent‚72.71177768707275 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 235.5703125# disk.dm-0.out‚55482.04052734375 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.drbd2265.in‚0 network.sent‚ 1.5153625e+07 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.373046875* proc.memory.percent‚18.063068389892578% disk./.usageGB‚281.90332412719727 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.73828125! proc.memory.rssMB‚ 2959.453125 cpu‚25.281396535103507 disk.sda.out‚ 235.5703125 disk.nvme0n1.in‚ 466.73828125 disk.sda2.out‚0 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚55482.04052734375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚18.918768614819832 disk.rbd1.in‚0 disk.sda3.out‚ 235.5703125) proc.memory.availableMB‚ 4470.90234375#Žø2¦- _wandbruntime‚6248  _runtime‚6248t¤)ɧà ' lr-AdamW/pg1‚0.00011555192978884664 ' lr-AdamW/pg2‚0.00011555192978884664 trainer/global_step‚6199 % _timestamp‚1.7884342706394186e+09 _runtime‚6248.994519364 _step‚507O@›OÎ¨È ' lr-AdamW/pg1‚0.00011555192978884664 ' lr-AdamW/pg2‚0.00011555192978884664 trainer/global_step‚6199 % _timestamp‚1.7884342706394186e+09 _runtime‚6248.994519364 _step‚507ûÆ"u+2©- _wandbruntime‚6248  _runtime‚6248SÅ€…(ªŠP N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115365389 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 % _timestamp‚1.7884342706402152e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133621844 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚856782815642112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13233882784698 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244926019 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843526506905600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚739109703335936 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18391702163968 _step‚508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133598738 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚856782815642112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66616765 trainer/global_step‚6199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13256308736512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842291811037758 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚856779683856896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48749165 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚856779683856896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚843526506905600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚739109786484736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720718056186368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18391647149568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13230779315774 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13256308736512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13253203342848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111327281 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133598738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48748627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244949131 _runtime‚6248.995620483 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133621844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66616762 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842294939934650 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚829061057149952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚843526480514048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚720718084320768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚829061031721984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚13028»Ù(«P GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244949131 _runtime‚6248.995620483 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133621844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66616762 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842294939934650 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚829061057149952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚843526480514048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚720718084320768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚829061031721984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115365389 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 % _timestamp‚1.7884342706402152e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133621844 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚856782815642112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13233882784698 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244926019 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843526506905600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚739109703335936 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18391702163968 _step‚508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133598738 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚856782815642112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66616765 trainer/global_step‚6199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13256308736512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842291811037758 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚856779683856896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48749165 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚856779683856896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚843526506905600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚739109786484736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720718056186368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18391647149568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13230779315774 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13256308736512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13253203342848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111327281 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133598738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48748627üŠ°¾æ2¬- _wandbruntime‚6249  _runtime‚6249–r­’N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720830467050496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244964393 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚739225070660608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720830495184896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚856916422057472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13258421063680 _step‚509 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚856916422057472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66627015 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚6249.888130895 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111344571 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843658000993792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚843657974602240 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111344565 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829190454151168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13232888108874 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244964393 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133619828 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13255315656704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111344571 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244987531 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66627018 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚842423316832074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133619828 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13258421063680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829190428723200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78843427153261e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_endÒçƒ@ÿ /inactive_split.small_pool.allocated‚48756752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18394658611200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚843657974602240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚843658000993792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115383254 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133642960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111344565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48756239 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244987531 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115383770YИ3'®—N _runtime‚6249.888130895 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111344571 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843658000993792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚843657974602240 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111344565 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829190454151168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13232888108874 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244964393 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133619828 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13255315656704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111344571 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244987531 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66627018 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚842423316832074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133619828 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13258421063680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829190428723200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78843427153261e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48756752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18394658611200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚843657974602240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚843658000993792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115383254 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133642960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111344565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48756239 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244987531 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115383770 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720830467050496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244964393 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚739225070660608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720830495184896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚856916422057472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13258421063680 _step‚509 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚856916422057472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66627015 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984ýþÇ'¬2¯- _wandbruntime‚6249  _runtime‚6249±½~%\°Ö epoch‚5 /loss/FRt2/detection_loss‚2.3527009013024243 0loss/FRt2/assignment_loss‚3.5630060894922777 0loss/FRt3/assignment_loss‚3.0510946715750347 2loss/SRqqt1/assignment_loss‚1.1761511241332914 /loss/FBt2/detection_loss‚2.5588209331035614 1loss/SRqqt2/assignment_loss‚2.117866104873626 /loss/FBt1/detection_loss‚0.7755374668717936 /loss/FBt3/detection_loss‚2.5265414227138865 0loss/FRt1/assignment_loss‚1.9240247985007057 %loss/total_loss‚45.17924710639102 _step‚510 % _timestamp‚1.7884342715343747e+09 1loss/FBt2/assignment_loss‚0.08935581168837169 .loss/FRt1/detection_loss‚1.607931148628114 1loss/SRqqt3/detection_loss‚1.9728932400544485 /loss/FRt4/assignment_loss‚4.416935176560373 1loss/SRqqt3/assignment_loss‚1.286974280079206 1loss/FBt1/assignment_loss‚0.07933497976378984 .loss/FRt3/detection_loss‚2.429794497606231 .loss/FRt4/detection_loss‚2.884585319143353 1loss/SRqqt2/detection_loss‚2.5955576145130657 1loss/SRqqt4/detection_loss‚2.3279198427354135 trainer/global_step‚6199 2loss/SRqqt4/assignment_loss‚1.5223870306245741 1loss/FBt3/assignment_loss‚0.16026485582055533 1loss/FBt4/assignment_loss‚0.09947682527490542 1loss/SRqqt1/detection_loss‚1.0046508030167647 .loss/FBt4/detection_loss‚2.655442168315252 _runtime‚6249.889401564 sÁLa±Û /loss/FBt1/detection_loss‚0.7755374668717936 /loss/FBt3/detection_loss‚2.5265414227138865 0loss/FRt1/assignment_loss‚1.9240247985007057 %loss/total_loss‚45.17924710639102 _step‚510 % _timestamp‚1.7884342715343747e+09 1loss/FBt2/assignment_loss‚0.08935581168837169 .loss/FRt1/detection_loss‚1.607931148628114 1loss/SRqqt3/detection_loss‚1.9728932400544485 /loss/FRt4/assignment_loss‚4.416935176560373 1loss/SRqqt3/assignment_loss‚1.286974280079206 1loss/FBt1/assignment_loss‚0.07933497976378984 .loss/FRt3/detection_loss‚2.429794497606231 .loss/FRt4/detection_loss‚2.884585319143353 1loss/SRqqt2/detection_loss‚2.5955576145130657 1loss/SRqqt4/detection_loss‚2.3279198427354135 trainer/global_step‚6199 2loss/SRqqt4/assignment_loss‚1.5223870306245741 1loss/FBt3/assignment_loss‚0.16026485582055533 1loss/FBt4/assignment_loss‚0.09947682527490542 1loss/SRqqt1/detection_loss‚1.0046508030167647 .loss/FBt4/detection_loss‚2.655442168315252 _runtime‚6249.889401564 epoch‚5 /loss/FRt2/detection_loss‚2.3527009013024243 0loss/FRt2/assignment_loss‚3.5630060894922777 0loss/FRt3/assignment_loss‚3.0510946715750347 2loss/SRqqt1/assignment_loss‚1.1761511241332914 /loss/FBt2/detection_loss‚2.5588209331035614 1loss/SRqqt2/assignment_loss‚2.117866104873626þ–€,†²‚(:’ ä®åÔ®àð¶ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚267.819' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.56380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ŸE_{³‚(:ð ä®åÔ—äÛ¸ disk.sda.in‚0.203125 network.recv‚ 2.129233e+06) disk./.usagePercent‚18.91997543433624 disk.drbd2266.out‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 4471.36328125& disk.nvme0n1.out‚55543.01708984375 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578" disk.nvme0c0n1.in‚ 466.73828125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 236.01953125 cpu‚25.280835590467763 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.373046875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 466.73828125# disk.dm-0.out‚55543.01708984375 disk.sda1.out‚0 disk.drbd1716.in‚0$ disk./.usageGB‚281.9213066101074( disk.nvme0c0n1.out‚55543.01708984375 disk.drbd2080.in‚0 disk.drbd2265.in‚0$ memory_percent‚72.70896434783936 network.sent‚ 1.5157153e+07 disk.sda2.in‚0 disk.dm-1.out‚ 236.01953125 disk.sda3.in‚0.203125 disk.sda.out‚ 236.01953125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.sda1.in‚0Uœ'»ž´‚(:“ ó®åÔлí´ gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚305.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.093399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KîÝ£vµ‚(:ë ó®åÔߧƷ disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 236.6328125! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578 disk.sda1.out‚0 network.sent‚ 1.5263161e+07 disk.rbd1.in‚0$ memory_percent‚72.71053791046143 disk.rbd1.out‚ 3.373046875 disk.dm-0.in‚ 466.73828125 disk.drbd2080.out‚0 disk.sda2.in‚0) disk./.usagePercent‚18.92112388413484 disk.sda3.in‚0.203125 disk.sda3.out‚ 236.6328125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 236.6328125& disk.nvme0n1.out‚55587.76318359375) proc.memory.availableMB‚ 4471.10546875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.sda2.out‚0( disk.nvme0c0n1.out‚55587.76318359375# disk.dm-0.out‚55587.76318359375 disk.drbd1716.in‚0 cpu‚25.280295511174437 network.recv‚ 2.13403e+06# disk./.usageGB‚281.938419342041 disk.drbd1716.out‚0Çav“¶‚(:ˆ ‚¯åÔÛ²µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚308.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.6268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã÷ñwx·‚(:í ‚¯åÔðÕ¤· disk.sda.in‚0.203125( proc.memory.availableMB‚ 4471.1015625* disk./.usagePercent‚18.922067271898516 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 236.9765625 disk.drbd2265.out‚0 disk.rbd0.in‚0% disk./.usageGB‚281.95247650146484( disk.nvme0c0n1.out‚55638.01318359375 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.sda.out‚ 236.9765625 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd2265.in‚0 cpu‚25.27971059235347 proc.cpu.threads‚17& disk.nvme0n1.out‚55638.01318359375 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 network.recv‚ 2.136133e+06 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 236.9765625 disk.rbd0.out‚0 network.sent‚ 1.5267228e+07 disk.sda2.out‚0$ memory_percent‚72.71056175231934 disk.nvme0n1.in‚ 466.73828125# disk.dm-0.out‚55638.01318359375 disk.sda1.out‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578 disk.dm-0.in‚ 466.73828125ÖG­ß2¸- _wandbruntime‚6296  _runtime‚6296¢Hɹà ' lr-AdamW/pg1‚0.00011408646773190155 ' lr-AdamW/pg2‚0.00011408646773190155 trainer/global_step‚6249 % _timestamp‚1.7884343178308613e+09 _runtime‚6296.185882305 _step‚511éæ|×ÎºÈ ' lr-AdamW/pg1‚0.00011408646773190155 ' lr-AdamW/pg2‚0.00011408646773190155 trainer/global_step‚6249 % _timestamp‚1.7884343178308613e+09 _runtime‚6296.185882305 _step‚511ÿ¢04•2»- _wandbruntime‚6296  _runtime‚6296¿te(¼‰P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134656590 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚726338627527168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚835530881782784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13361938843136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚850101211315200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134679696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863460018373120 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116259729 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13358833449472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49130317 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246848071 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744877287903232 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚850101184923648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13339336030906 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246871183 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67129412 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863460018373120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚848867114344766 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚850101211315200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134679696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112191481 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116260268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚726338599392768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚848870243241658 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚850101184923648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112191487 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744877371052032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112191487 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚863463150158336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18538688510464 $ _timestamp‚1.788434317831781e+09 _step‚512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13361938843136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13358833449472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49130853 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18538743524864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13336232561982 _runtime‚6296.187118408Z U=½ŽP N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116259729 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13358833449472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49130317 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246848071 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744877287903232 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \HDeviceStatsMon¥½B×#itor.on_train_batch_start/allocated_bytes.large_pool.freed‚850101184923648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13339336030906 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246871183 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67129412 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863460018373120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚848867114344766 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚850101211315200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134679696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112191481 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116260268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚726338599392768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚848870243241658 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚850101184923648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112191487 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744877371052032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112191487 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚863463150158336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18538688510464 $ _timestamp‚1.788434317831781e+09 _step‚512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13361938843136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13358833449472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49130853 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18538743524864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13336232561982 _runtime‚6296.187118408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134656590 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚726338627527168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚835530881782784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13361938843136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚850101211315200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134679696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863460018373120€î¢ÿX2¾- _wandbruntime‚6297  _runtime‚6297±É˜'¿’N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246886445 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863596756573696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13364051170304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49138507 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13364051170304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚850232705403392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863593624775168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18541638680576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112208765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67139668 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚848998620139082 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49137988 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112208771 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246909583 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 # _timestamp‚1.78843431873368e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚513 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚726451038391296 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚850232705403392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863596756573696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112208771 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13360945763328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112208765 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13341444824214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚726451010256896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚744992648937472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116277653 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134677680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13360945763328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚835660278784000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67139665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246909583 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116278175 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13338341355082 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246886445 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚850232679011840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49137988 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112208771 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246909583 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 # _timestamp‚1.78843431873368e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚513 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚726451038391296 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚850232705403392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863596756573696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112208771 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13360945763328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112208765 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13341444824214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚726451010256896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚744992648937472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116277653 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134677680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13360945763328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚835660278784000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67139665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246909583 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116278175 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13338341355082 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246886445 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚850232679011840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18685669729792 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117152278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚6343.868738365 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚6299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113055687 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚856675915724800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚856675915724800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117152821 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚750644757314560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113055681 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135737548 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13464463556096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚870143484674560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚870140352889344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚856675889333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135714442 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚855445546548666 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13467568949760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 % _timestamp‚1.7884343655134559e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248770123 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13441685808190 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _step‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113055681 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18685614715392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113055687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67642065 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135714442 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13444789277114 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13464463556096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49510216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67642062 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13467568949760 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚870143484674560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855442417651774 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚870140352889344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135737548 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49510756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚731959170733568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚731959142599168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚856675889333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚750644840463360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚842000757271552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¨·P×(ÏP UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113055687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67642065 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135714442 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13444789277114 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13464463556096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49510216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67642062 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13467568949760 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚870143484674560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855442417651774 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚870140352889344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135737548 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49510756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚731959170733568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚731959142599168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚856675889333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚750644840463360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚842000757271552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248793235 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚842000731843584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18685669729792 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117152278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚6343.868738365 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚6299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113055687 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚856675915724800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚856675915724800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117152821 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚750644757314560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113055681 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135737548 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13464463556096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚870143484674560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚870140352889344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚856675889333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135714442 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚855445546548666 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13467568949760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 % _timestamp‚1.7884343655134559e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248770123 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13441685808190 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _step‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113055681 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18685614715392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180„ }´2Ð- _wandbruntime‚6344  _runtime‚6344ötäÔÙÑ”N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚732071581597696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750760109956096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚732071553463296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13469681276928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750760193091584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚855573923446090 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113072971 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚856807383421440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884343663972464e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚856807383421440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13466575869952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49518357 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13443794601290 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117170675 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13446898070422 _runtime‚6344.752978911 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚856807409812992 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚842130128844800 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135735532 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚870277091089920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚870273959291392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49517846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248808497 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13466575869952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 trainer/global_step‚6299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚870277091089920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248831635 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248808497 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113072971 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135758664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135735532 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚842130154272768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117170161 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302¬s;§'Ò™N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚732071581597696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750760109956096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚732071553463296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13469681276928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750760193091584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚855573923446090 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113072971 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚856807383421440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884343663972464e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚856807383421440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13466575869952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49518357 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13443794601290 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117170675 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13446898070422 _runtime‚6344.752978911 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚856807409812992 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚842130128844800 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135735532 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚870277091089920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚870273959291392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49517846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248808497 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13466575869952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 trainer/global_step‚6299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚870277091089920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248831635 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248808497 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113072971 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135758664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135735532 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚842130154272768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117170161 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878…yqäþ2Ó- _wandbruntime‚6344  _runtime‚6344¬gç]Ô× 2loss/SRqqt3/assignment_loss‚1.2957158997993579 /loss/FRt2/detection_loss‚2.4842713041150053 % _timestamp‚1.7884343663990793e+09 0loss/FRt2/assignment_loss‚3.2836276461248812 .loss/FBt4/detection_loss‚2.415410804748535 trainer/global_step‚6299 _runtime‚ 6344.75402743 2loss/SRqqt4/assignment_loss‚1.7757624305527786 0loss/SRqqt1/detection_loss‚1.030690741790242 1loss/SRqqt2/detection_loss‚2.2335424853695764 0loss/SRqqt4/detection_loss‚2.366549078760476 /loss/FRt1/detection_loss‚1.4996352016393628 /loss/FRt4/assignment_loss‚3.920203369239281 2loss/FBt1/assignment_loss‚0.029017200012591744 epoch‚5 /loss/FRt4/detection_loss‚3.0229808749823737 %loss/total_loss‚43.05729977078509 1loss/FBt4/assignment_loss‚0.09652672314162297 0loss/FRt1/assignment_loss‚1.8626694713320051 /loss/FRt3/assignment_loss‚2.978415069480737 2loss/SRqqt1/assignment_loss‚1.1914111184346121 1loss/FBt2/assignment_loss‚0.23300890689242237 /loss/FRt3/detection_loss‚2.3614798560738564 1loss/SRqqt3/detection_loss‚2.1748841526833447 /loss/FBt1/detection_loss‚0.4930066685837049 .loss/FBt2/detection_loss‚2.146437394618988 _step‚518 1loss/FBt3/assignment_loss‚0.25062934925836045 /loss/FBt3/detection_loss‚2.1410773284733295 2loss/SRqqt2/assignment_loss‚1.7703466946776543ÕCg7bÕÜ 2loss/SRqqt4/assignment_loss‚1.7757624305527786 0loss/SRqqt1/detection_loss‚1.030690741790242 1loss/SRqqt2/detection_loss‚2.2335424853695764 0loss/SRqqt4/detection_loss‚2.366549078760476 /loss/FRt1/detection_loss‚1.4996352016393628 /loss/FRt4/assignment_loss‚3.920203369239281 2loss/FBt1/assignment_loss‚0.029017200012591744 epoch‚5 /loss/FRt4/detection_loss‚3.0229808749823737 %loss/total_loss‚43.05729977078509 1loss/FBt4/assignment_loss‚0.09652672314162297 0loss/FRt1/assignment_loss‚1.8626694713320051 /loss/FRt3/assignment_loss‚2.978415069480737 2loss/SRqqt1/assignment_loss‚1.1914111184346121 1loss/FBt2/assignment_loss‚0.23300890689242237 /loss/FRt3/detection_loss‚2.3614798560738564 1loss/SRqqt3/detection_loss‚2.1748841526833447 /loss/FBt1/detection_loss‚0.4930066685837049 .loss/FBt2/detection_loss‚2.146437394618988 _step‚518 1loss/FBt3/assignment_loss‚0.25062934925836045 /loss/FBt3/detection_loss‚2.1410773284733295 2loss/SRqqt2/assignment_loss‚1.7703466946776543 2loss/SRqqt3/assignment_loss‚1.2957158997993579 /loss/FRt2/detection_loss‚2.4842713041150053 % _timestamp‚1.7884343663990793e+09 0loss/FRt2/assignment_loss‚3.2836276461248812 .loss/FBt4/detection_loss‚2.415410804748535 trainer/global_step‚6299 _runtime‚ 6344.75402743†òô òžÖ‚(:“ ¾¯åԃݧµ gpu.0.gpu‚49.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚306.642' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.328399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë%–†lׂ(:á ¾¯åÔ„äç·$ disk./.usageGB‚282.0198097229004 disk.sda1.in‚0 disk.drbd2265.out‚0$ memory_percent‚99.57561492919922* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 disk.dm-1.out‚ 239.41796875" disk.nvme0c0n1.in‚ 466.73828125% disk.nvme0c0n1.out‚55900.00390625 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.dm-0.in‚ 466.73828125 disk.sda.out‚ 239.41796875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 network.recv‚ 2.150832e+06) disk./.usagePercent‚18.92658606088539 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.sda2.in‚0 disk.drbd2265.in‚0$ proc.memory.availableMB‚69.53125 cpu‚25.277542919291275 disk.drbd2080.out‚0 disk.sda3.out‚ 239.41796875! proc.memory.rssMB‚ 2959.453125 disk.dm-1.in‚0.203125 disk.drbd1716.out‚0 network.sent‚ 1.538732e+07# disk.nvme0n1.out‚55900.00390625 disk.sda2.out‚0 disk.rbd1.out‚ 3.373046875 disk.dm-0.out‚55900.00390625 disk.sda3.in‚0.203125 disk.rbd0.in‚0A¦ði“Ø‚(:ˆ ͯåÔ³Ôƒµ gpu.0.gpu‚40.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚126.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a•üùrÙ‚(:ç ͯåÔ¥ó…· disk.dm-1.out‚ 240.03125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0$ memory_percent‚99.57554340362549% disk./.usageGB‚282.04187393188477 disk.rbd1.out‚ 3.373046875 disk.rbd0.out‚0 network.recv‚ 2.156786e+06 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.73828125 disk.rbd0.in‚0' proc.memory.availableMB‚ 69.54296875! proc.memory.rssMB‚ 2959.453125 disk.sda3.out‚ 240.03125 disk.sda1.out‚0# disk.dm-0.out‚55960.60888671875 disk.sda3.in‚0.203125 disk.nvme0n1.in‚ 466.73828125 disk.dm-1.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.277014650230793 disk.drbd1716.in‚0 disk.sda.out‚ 240.03125* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17* disk./.usagePercent‚18.928066808463438 network.sent‚ 1.5494539e+07& disk.nvme0n1.out‚55960.60888671875 disk.sda2.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.73828125( disk.nvme0c0n1.out‚55960.60888671875 disk.sda2.out‚0Ì «Ú“Ú‚(:ˆ ܯåÔ·—þ´ gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ă–wÛ‚(:ì ܯåÔ´Í· disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 69.5390625* disk./.usagePercent‚18.929899822408597 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚56034.59423828125 disk.sda3.out‚ 240.44140625 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.73828125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 240.44140625 disk.sda.out‚ 240.44140625! proc.memory.rssMB‚ 2959.453125 disk.rbd1.in‚0* proc.memory.percent‚18.063068389892578 disk.drbd1716.out‚0 disk.drbd2080.out‚0 network.recv‚ 2.15924e+06 disk.rbd1.out‚ 3.373046875 disk.dm-0.in‚ 466.73828125 cpu‚25.27647816310399 network.sent‚ 1.5498199e+07% disk./.usageGB‚282.06918716430664 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0# memory_percent‚99.5755672454834 disk.sda.in‚0.203125" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd1716.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚56034.59423828125 disk.sda2.in‚0 disk.drbd2266.in‚0# disk.dm-0.out‚56034.59423828125 disk.sda3.in‚0.203125dâ˞܂(:“ ë¯åÔÉ¿—µ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚290.315' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.062999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j•bCrÝ‚(:ç ë¯åÔ¬¢Õ¶ disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125& disk.nvme0n1.out‚56103.51123046875 disk.dm-1.out‚ 241.203125 disk.drbd2266.in‚0# disk.dm-0.out‚56103.51123046875 disk.sda.out‚ 241.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 466.73828125 disk.sda3.out‚ 241.203125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.rbd1.in‚0( disk.nvme0c0n1.out‚56103.51123046875) disk./.usagePercent‚18.93170646757773 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.sda2.in‚0$ memory_percent‚99.57709312438965 network.sent‚ 1.5502253e+07 disk.drbd1716.in‚0 disk.drbd2266.out‚0 network.recv‚ 2.161572e+06 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 3.373046875" disk.nvme0c0n1.in‚ 466.73828125& proc.memory.availableMB‚ 69.2890625* proc.memory.percent‚18.063068389892578 cpu‚25.27599522991282 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚0.203125! proc.memory.rssMB‚ 2959.453125% disk./.usageGB‚282.09610748291016uQ7Æ2Þ- _wandbruntime‚6391  _runtime‚6391Ï×/Èß $ _timestamp‚1.788434413162857e+09 _runtime‚6391.517900512 _step‚519 ' lr-AdamW/pg1‚0.00011114847143942972 ' lr-AdamW/pg2‚0.00011114847143942972 trainer/global_step‚6349Æ\CÍàÇ ' lr-AdamW/pg1‚0.00011114847143942972 ' lr-AdamW/pg2‚0.00011114847143942972 trainer/global_step‚6349 $ _timestamp‚1.788434413162857e+09 _runtime‚6391.517900512 _step‚519‡VŸ 2á- _wandbruntime‚6391  _runtime‚6391·àvd(âŠP VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚876823819190784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 % _timestamp‚1.7884344131635993e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118045186 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚863250620134400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18832559364608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚863250593742848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚737579685805568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚862020849855674 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚863250593742848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13547139054398 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113919881 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136772294 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68154715 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756412188058624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18832502253056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚862017720958782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49890474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚863250620134400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚876820687405568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136772294 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 trainer/global_step‚6349 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68154715 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756412188058624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18832502253056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚862017720958782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49890474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚863250620134400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚876820687405568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136772294 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 trainer/global_step‚6349 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118045186 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚863250620134400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18832559364608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚863250593742848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚737579685805568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚862020849855674 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚863250593742848ˆºæ2ä- _wandbruntime‚6392  _runtime‚6392Ó'P'å”N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136793384 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136816516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250753687 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118063176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13575311383552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚863382087831040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚737692096669696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18835478375936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13552351316630 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚6349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250730549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49898727 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚876954293807616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13572205976576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49898211 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884344140655591e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136816516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚863382114222592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250730549 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118063695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113937165 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250753687 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13549247847498 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚6392.421172086 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113937171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329FÒ£'æ™N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68164968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884344140655591e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136816516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚863382114222592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250730549 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118063695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113937165 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250753687 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13549247847498 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚6392.421172086 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113937171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136793384 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136816516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250753687 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118063176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13575311383552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚863382087831040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚737692096669696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18835478375936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13552351316630 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚6349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250730549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49898727 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚876954293807616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13572205976576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49898211 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚883501021921792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68667365 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252637339 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137853252 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚869825298152448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118940765 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884344605557594e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883504153707008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68667362 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13678829163008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137830146 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚762179556632576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883501021921792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13675723769344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883504153707008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137830146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968xe+(õP TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50273403 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137853252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114784087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13655695769530 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13652592300606 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚869825324544000 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚6438.911031842 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚883501021921792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68667365 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252637339 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137853252 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚869825298152448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118940765 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884344605557594e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883504153707008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68667362 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13678829163008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137830146 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚762179556632576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883501021921792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13675723769344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883504153707008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137830146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚868596153162682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252614227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50273943 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚869825324544000Œ‘E2ö- _wandbruntime‚6439  _runtime‚6439Bì‡u÷”N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6399 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118958544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚883637760122368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6439.810187638 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137874368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13654701093706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13677836083200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚855069854394368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13680941490176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚869956792240640 % _timestamp‚1.7884344614541886e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚869956818632192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50281440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252652601 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868724530060106 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13680941490176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚762294914871296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚883637760122368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114801371 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18982327899136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18982274995200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚855069828966400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13657804562838 I@DeviceStatsMonitor.on_train_batch_end/segment.small€ó÷¥ _pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13677836083200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252675739 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137874368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114801365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50280929 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68677618 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚869956792240640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚869956818632192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137851236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚743312639876096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50281440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252652601 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868724530060106 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13680941490176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚762294914871296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚883637760122368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114801371 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18982327899136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18982274995200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚855069828966400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13657804562838 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13677836083200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252675739 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137874368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114801365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50280929 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68677618 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚869956792240640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚869956818632192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137851236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚743312639876096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6399 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118958544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚883637760122368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6439.810187638 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137874368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13654701093706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13677836083200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚855069854394368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13680941490176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514ñ¸þ2ù- _wandbruntime‚6439  _runtime‚6439Êè„P^úØ /loss/FRt4/detection_loss‚3.2546440155609795 0loss/FRt1/assignment_loss‚1.5174117818684645 0loss/FRt4/assignment_loss‚3.7558954928232278 /loss/FRt3/detection_loss‚2.3837313790890304 1loss/SRqqt1/detection_loss‚0.9938742251157068 /loss/FBt4/detection_loss‚1.5074499034219317 trainer/global_step‚6399 0loss/FRt3/assignment_loss‚2.9513403190807863 % _timestamp‚1.7884344614568539e+09 /loss/FRt2/assignment_loss‚3.474211483290701 2loss/SRqqt2/assignment_loss‚1.4902157964450973 0loss/SRqqt3/assignment_loss‚1.59037804223445 1loss/FBt2/assignment_loss‚0.08808648955002478 0loss/SRqqt4/detection_loss‚2.373516090891578 .loss/FBt2/detection_loss‚2.739788908097479 .loss/FBt3/detection_loss‚2.705523081123829 /loss/FRt1/detection_loss‚1.5316166114849104 1loss/SRqqt2/detection_loss‚2.4814837841050967 _runtime‚6439.811832874 _step‚526 /loss/FRt2/detection_loss‚2.4899599082542188 /loss/FBt1/detection_loss‚0.7073694220571606 %loss/total_loss‚43.21076342690173 epoch‚5 1loss/SRqqt3/detection_loss‚1.9709499338451697 2loss/SRqqt1/assignment_loss‚1.2523949033862283 2loss/SRqqt4/assignment_loss‚1.3870561077739254 2loss/FBt1/assignment_loss‚0.009837773421989429 1loss/FBt3/assignment_loss‚0.22039778763419804 1loss/FBt4/assignment_loss‚0.33363018634554464ߟò+cûÝ 0loss/FRt3/assignment_loss‚2.9513403190807863 % _timestamp‚1.7884344614568539e+09 /loss/FRt2/assignment_loss‚3.474211483290701 2loss/SRqqt2/assignment_loss‚1.4902157964450973 0loss/SRqqt3/assignment_loss‚1.59037804223445 1loss/FBt2/assignment_loss‚0.08808648955002478 0loss/SRqqt4/detection_loss‚2.373516090891578 .loss/FBt2/detection_loss‚2.739788908097479 .loss/FBt3/detection_loss‚2.705523081123829 /loss/FRt1/detection_loss‚1.5316166114849104 1loss/SRqqt2/detection_loss‚2.4814837841050967 _runtime‚6439.811832874 _step‚526 /loss/FRt2/detection_loss‚2.4899599082542188 /loss/FBt1/detection_loss‚0.7073694220571606 %loss/total_loss‚43.21076342690173 epoch‚5 1loss/SRqqt3/detection_loss‚1.9709499338451697 2loss/SRqqt1/assignment_loss‚1.2523949033862283 2loss/SRqqt4/assignment_loss‚1.3870561077739254 2loss/FBt1/assignment_loss‚0.009837773421989429 1loss/FBt3/assignment_loss‚0.22039778763419804 1loss/FBt4/assignment_loss‚0.33363018634554464 /loss/FRt4/detection_loss‚3.2546440155609795 0loss/FRt1/assignment_loss‚1.5174117818684645 0loss/FRt4/assignment_loss‚3.7558954928232278 /loss/FRt3/detection_loss‚2.3837313790890304 1loss/SRqqt1/detection_loss‚0.9938742251157068 /loss/FBt4/detection_loss‚1.5074499034219317 trainer/global_step‚6399ŽÇŠžü‚(:“ §°åÔƒ»”µ gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.202' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.440400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G÷ì vý‚(:ë §°åÔÿ‡ô¶ network.recv‚ 2.177359e+06 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚18.93817731401319 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.203125 network.sent‚ 1.5623944e+07$ disk./.usageGB‚282.1925277709961 disk.sda.out‚ 243.31640625 disk.rbd0.out‚0& disk.nvme0n1.out‚56374.01220703125 disk.rbd1.out‚ 3.373046875 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.dm-1.out‚ 243.31640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚56374.01220703125 disk.dm-0.in‚ 466.73828125 disk.sda3.out‚ 243.31640625 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578 disk.rbd0.in‚0# disk.dm-0.out‚56374.01220703125# memory_percent‚99.5725154876709& proc.memory.availableMB‚ 70.0390625 cpu‚25.27389932677478 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.73828125%·lžþ‚(:“ ¶°åÔþ³û´ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚286.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.239200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óà1yÿ‚(:î ¶°åÔä’Ä· disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.73828125 disk.drbd2080.out‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 70.04296875 cpu‚25.273376798891015 disk.dm-1.out‚ 243.81640625 disk.rbd0.out‚0 network.sent‚ 1.573085e+07% disk./.usageGB‚282.21099853515625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.373046875( disk.nvme0c0n1.out‚56440.52001953125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 243.81640625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.73828125# disk.dm-0.out‚56440.52001953125 disk.sda.out‚ 243.81640625$ memory_percent‚99.57249164581299 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125& disk.nvme0n1.out‚56440.52001953125! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.rbd0.in‚0 network.recv‚ 2.182442e+06) disk./.usagePercent‚18.93941690249398Œ¥¥ž€‚(:“ Å°åÔþ’¨µ gpu.0.gpu‚42.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.847' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.169399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K5Lz‚(:ï Å°åÔ¼¯‰· disk.sda2.out‚0$ memory_percent‚99.57249164581299 network.sent‚ 1.5735695e+07 disk.drbd2265.in‚0 disk.drbd2265.out‚0) disk./.usagePercent‚18.94005359323164 disk.sda3.in‚0.203125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.73828125 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 244.69140625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875" disk.nvme0c0n1.in‚ 466.73828125( disk.nvme0c0n1.out‚56490.90576171875 network.recv‚ 2.186148e+06 disk.sda3.out‚ 244.69140625 disk.sda1.in‚0 disk.dm-1.out‚ 244.69140625! proc.memory.rssMB‚ 2959.453125 cpu‚25.272865497023542% disk./.usageGB‚282.22048568725586* proc.memory.percent‚18.063068389892578 disk.drbd1716.out‚0 disk.sda.in‚0.203125# disk.dm-0.out‚56490.90576171875' proc.memory.availableMB‚ 70.04296875 disk.drbd2080.out‚0& disk.nvme0n1.out‚56490.90576171875Xh62‚- _wandbruntime‚6486  _runtime‚6486¬@õîȃ ' lr-AdamW/pg1‚0.00010820308717304302 ' lr-AdamW/pg2‚0.00010820308717304302 trainer/global_step‚6449 $ _timestamp‚1.788434508024286e+09 _runtime‚6486.379306855 _step‚527ŸM…×Í„Ç $ _timestamp‚1.788434508024286e+09 _runtime‚6486.379306855 _step‚527 ' lr-AdamW/pg1‚0.00010820308717304302 ' lr-AdamW/pg2‚0.00010820308717304302 trainer/global_step‚6449£©\2…- _wandbruntime‚6486  _runtime‚6486¯SnF(†ŠP @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884345080249577e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚890181356438016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876400028953600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115648281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _runtime‚6486.380245516 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876400028953600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138887998 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875168327572798 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚890184488223232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138911104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138887998 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚876400002562048 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13784459269632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115648287 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876400002562048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13784459269632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚890181356438016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748820800352768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254536279 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50656005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13781353875968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚767947197004800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875171456469690 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13758045546814 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254559391 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69180015 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119835452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚767947113856000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13761149015738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552’M†V†‡P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13781353875968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚767947197004800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875171456469690 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13758045546814 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254559391 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69180015 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119835452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚767947113856000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13761149015738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 % _timestamp‚1.7884345080249577e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚890181356438016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876400028953600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115648281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _runtime‚6486.380245516 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876400028953600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138887998 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875168327572798 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚890184488223232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138911104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138887998 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚876400002562048 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13784459269632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115648287 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876400002562048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13784459269632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚890181356438016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748820800352768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254536279 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50656005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ñVpr2ˆ- _wandbruntime‚6487  _runtime‚6487þjfo'‰“N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19129259814400 _step‚529 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876531523041792 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13783466189824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254597791 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69190268 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13760154339914 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748933211216896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚876531496650240 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13763257809046 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚ 6487.27971234 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254574653 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254597791 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119853206 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768062528129536 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50662941 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚890318094638592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚876531523041792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50663465 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119853733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 % _timestamp‚1.7884345089241943e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13786571596800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115665571 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254574653 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115665565 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 trainer/global_step‚6449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138932220 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976ºóé\'Š˜N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚876531496650240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚890318094638592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13783466189824 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138909088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254597791 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119853206 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768062528129536 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50662941 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚890318094638592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚876531523041792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50663465 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119853733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 % _timestamp‚1.7884345089241943e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13786571596800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115665571 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254574653 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115665565 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 trainer/global_step‚6449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138932220 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19129259814400 _step‚529 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876531523041792 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13783466189824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254597791 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69190268 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13760154339914 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748933211216896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚876531496650240 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13763257809046 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚ 6487.27971234 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254574653 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19272932858880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚881743630879806 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139945850 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116512487 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867880132086784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚882974706971648 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120729673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13866602261946 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13890089376256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 % _timestamp‚1.7884345557526433e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚754441343559168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139945850 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896861690954240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚754441315424768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚882974706971648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51037011 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13886983982592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚6499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚882974733363200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256481443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69692665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19272932858880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚881743630879806 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139945850 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116512487 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867880132086784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚882974706971648 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120729673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13866602261946 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13890089376256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 % _timestamp‚1.7884345557526433e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚754441343559168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139945850 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896861690954240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚754441315424768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚882974706971648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51037011 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13886983982592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚6499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚882974733363200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256481443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69692665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120748202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚868009554515968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6499 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69702915 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139990072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13889096296448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13892201703424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116529771 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚881875136674122 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚883106227451392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116529765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51045284 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884345566503515e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116529771 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚896998429154816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚754553754423296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51044765 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19275798965760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚896995297356288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13889096296448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19275853966848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚883106201059840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139966940 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚868009529088000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚øwÞ& 120747680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13865607586122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256519843 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256496705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69702918 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13868711055254 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13892201703424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120747680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13865607586122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256519843 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256496705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69702918 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13868711055254 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13892201703424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120748202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚868009554515968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6499 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69702915 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139990072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13889096296448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13892201703424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116529771 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚881875136674122 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚883106227451392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116529765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51045284 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884345566503515e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116529771 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚896998429154816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚754553754423296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51044765 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19275798965760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚896995297356288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13889096296448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19275853966848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456•XVº2- _wandbruntime‚6535  _runtime‚6535‡®"´ZžÔ .loss/FRt2/detection_loss‚2.387650663076445 2loss/SRqqt1/assignment_loss‚0.9033646063943361 .loss/FRt4/detection_loss‚3.081140003822468 0loss/FRt2/assignment_loss‚3.8658972493437833 0loss/FRt3/assignment_loss‚2.5565878632995815 2loss/SRqqt2/assignment_loss‚1.5280626107835107 0loss/FBt4/assignment_loss‚0.2092500075454462 0loss/FRt4/assignment_loss‚3.2952863066284745 0loss/SRqqt2/detection_loss‚2.216218249665366 1loss/SRqqt3/detection_loss‚1.5964406314823363 0loss/SRqqt4/detection_loss‚2.611289075829766 /loss/FBt1/detection_loss‚0.8309473220879833 .loss/FBt4/detection_loss‚2.173176477937137 1loss/FBt3/assignment_loss‚0.14332321488901095 1loss/SRqqt1/detection_loss‚0.9853523735856188 .loss/FBt2/detection_loss‚2.459272330457514 .loss/FBt3/detection_loss‚1.878536646856981 % _timestamp‚1.7884345566522021e+09 _runtime‚6535.007206934 _step‚534 %loss/total_loss‚41.82066446076385 trainer/global_step‚6499 0loss/FRt1/assignment_loss‚1.3220494233644926 2loss/SRqqt3/assignment_loss‚1.5332517739799287 1loss/FBt2/assignment_loss‚0.39755637833679264 .loss/FRt1/detection_loss‚1.502476023653379 /loss/FRt3/detection_loss‚2.6591233991914325 1loss/SRqqt4/assignment_loss‚1.677696754309264 2loss/FBt1/assignment_loss‚0.006715074242798854 epoch‚5Ä£Ñ_ŸÙ 0loss/FRt3/assignment_loss‚2.5565878632995815 2loss/SRqqt2/assignment_loss‚1.5280626107835107 0loss/FBt4/assignment_loss‚0.2092500075454462 0loss/FRt4/assignment_loss‚3.2952863066284745 0loss/SRqqt2/detection_loss‚2.216218249665366 1loss/SRqqt3/detection_loss‚1.5964406314823363 0loss/SRqqt4/detection_loss‚2.611289075829766 /loss/FBt1/detection_loss‚0.8309473220879833 .loss/FBt4/detection_loss‚2.173176477937137 1loss/FBt3/assignment_loss‚0.14332321488901095 1loss/SRqqt1/detection_loss‚0.9853523735856188 .loss/FBt2/detection_loss‚2.459272330457514 .loss/FBt3/detection_loss‚1.878536646856981 % _timestamp‚1.7884345566522021e+09 _runtime‚6535.007206934 _step‚534 %loss/total_loss‚41.82066446076385 trainer/global_step‚6499 0loss/FRt1/assignment_loss‚1.3220494233644926 2loss/SRqqt3/assignment_loss‚1.5332517739799287 1loss/FBt2/assignment_loss‚0.39755637833679264 .loss/FRt1/detection_loss‚1.502476023653379 /loss/FRt3/detection_loss‚2.6591233991914325 1loss/SRqqt4/assignment_loss‚1.677696754309264 2loss/FBt1/assignment_loss‚0.006715074242798854 epoch‚5 .loss/FRt2/detection_loss‚2.387650663076445 2loss/SRqqt1/assignment_loss‚0.9033646063943361 .loss/FRt4/detection_loss‚3.081140003822468 0loss/FRt2/assignment_loss‚3.8658972493437833–"§-i“ ‚(:ˆ ±åԎŵ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[_Õ·w¡‚(:ì ±åÔºéó¶ disk.dm-1.in‚0.203125' proc.memory.availableMB‚ 69.94140625 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda3.out‚ 246.61328125 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2959.453125 disk.nvme0n1.in‚ 466.73828125 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚56758.67919921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 246.61328125 disk.dm-1.out‚ 246.61328125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0$ memory_percent‚99.57311153411865 network.recv‚ 2.198373e+06# disk./.usageGB‚282.310733795166 disk.rbd0.in‚0* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17& disk.nvme0n1.out‚56758.67919921875( disk.nvme0c0n1.out‚56758.67919921875 disk.sda3.in‚0.203125 cpu‚25.270792177534386" disk.nvme0c0n1.in‚ 466.73828125 disk.dm-0.in‚ 466.73828125 disk.drbd2080.out‚0 network.sent‚ 1.585476e+07) disk./.usagePercent‚18.94611021947669 disk.rbd0.out‚0ÙLÕ梂(:’ ±åÔð©Ïµ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AC‹l£‚(:á ±åÔÑàü·* proc.memory.percent‚18.063068389892578 cpu‚25.270280314768936 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.73828125 network.recv‚ 2.202197e+06 disk.sda2.out‚0 disk.dm-1.out‚247.0625 disk.rbd1.out‚ 3.373046875 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.73828125 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚56830.7724609375" disk.dm-0.out‚56830.7724609375% disk./.usageGB‚282.33866119384766' proc.memory.availableMB‚ 69.94140625 proc.cpu.threads‚17 disk.sda3.out‚247.0625 disk.drbd1716.out‚0% disk.nvme0n1.out‚56830.7724609375 disk.sda2.in‚0 disk.drbd2266.out‚0$ memory_percent‚99.57311153411865 network.sent‚ 1.5960311e+07 disk.sda.out‚247.0625 disk.rbd1.in‚0* disk./.usagePercent‚18.947984450634866 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125! proc.memory.rssMB‚ 2959.453125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.73828125ñlt-ž¤‚(:“ Ÿ±åÔš¼ð´ gpu.0.gpu‚62.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚272.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.485400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Û+¤w¥‚(:ì Ÿ±åԇȷ disk.sda2.in‚0" disk.dm-0.out‚56904.8212890625 disk.drbd1716.in‚0 disk.sda.out‚ 247.75390625 cpu‚25.26982064841707 network.sent‚ 1.5966522e+07 disk.dm-1.in‚0.203125 disk.drbd2080.out‚0 network.recv‚ 2.206116e+06 disk.rbd0.in‚0% disk.nvme0n1.out‚56904.8212890625 disk.drbd2080.in‚0 disk.sda.in‚0.203125 proc.cpu.threads‚17* disk./.usagePercent‚18.949228135236204 disk.drbd2265.out‚0 disk.dm-1.out‚ 247.75390625" disk.nvme0c0n1.in‚ 466.73828125 disk.dm-0.in‚ 466.73828125 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.063068389892578% disk./.usageGB‚282.35719299316406 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.73828125 disk.rbd1.out‚ 3.373046875' proc.memory.availableMB‚ 69.43359375! proc.memory.rssMB‚ 2959.453125 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 247.75390625 disk.drbd2265.in‚0$ memory_percent‚99.57621097564697 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚56904.8212890625 disk.sda1.in‚0ÛA¡á2¦- _wandbruntime‚6582  _runtime‚6582·òȧ ' lr-AdamW/pg1‚0.00010525277005948944 ' lr-AdamW/pg2‚0.00010525277005948944 trainer/global_step‚6549 % _timestamp‚1.7884346036852653e+09 _runtime‚ 6582.04029355 _step‚535ž? Í¨Ç _step‚535 ' lr-AdamW/pg1‚0.00010525277005948944 ' lr-AdamW/pg2‚0.00010525277005948944 trainer/global_step‚6549 % _timestamp‚1.7884346036852653e+09 _runtime‚ 6582.04029355—>í,Z2©- _wandbruntime‚6582  _runtime‚6582Úš;(ª‰P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚903542025470464 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70205315 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13968952039230 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚889549411381248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚889549411381248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874350007575552 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13995719482880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141003702 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141003702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚760061858631168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚779481477475840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117376681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258403495 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121623889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51419117 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚6549 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117376681 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 J) «ŽP SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚903542025470464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117376687 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141026808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13972055508154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258380383 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 _step‚536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚903545157255680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚903542025470464 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70205315 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13968952039230 D:DevöŒÝAëiceStatsMonitor.on_train_batch_start/active.all.current‚23112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚889549411381248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚889549411381248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874350007575552 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13995719482880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141003702 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141003702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚760061858631168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚779481477475840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117376681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258403495 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121623889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51419117 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚6549 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117376681 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121641856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70215565 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚903678763671040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13997831810048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13971060832330 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51426291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19422548028928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258441895 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258441895 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13994726403072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258418757 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117393971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚889680931860992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258418757 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874479379148800 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13994726403072 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚889680905469440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70215568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51426825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚779596902756864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117393965 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878±@°p'®™N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚889680931860992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚779596817524224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19422605127168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121641856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70215565 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚903678763671040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13997831810048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13971060832330 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51426291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19422548028928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258441895 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258441895 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13994726403072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258418757 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117393971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚889680931860992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258418757 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874479379148800 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13994726403072 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚889680905469440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70215568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51426825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚779596902756864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117393965 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118240887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910222359986688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14074405285438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚910222359986688 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260325547 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚880819832208384 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 •(¿ŽP [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14074405285438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚910222359986688 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260325547 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚880819832208384 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14101349589504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118240887 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚910225491771904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚785248714421248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚880819857636352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70717965 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142061554 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚765682429971968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118240887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910222359986688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552œX<32À- _wandbruntime‚6630  _runtime‚6630™š·g²Á”N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚910359098187264 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260340809 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚785364072104448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896255609879040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118258165 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19569178364416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142105776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 _runtime‚6630.921680972 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚910359098187264 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14100356509696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚765794812701696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚880949254637568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14076514078538 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260363947 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51808624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚765794840836096 trainer/global_step‚6599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896255636270592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚896255609879040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14103461916672 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122536842 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚896255636270592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚895028872185238 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260340809 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70728215 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884346525659473e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_spá3-h lit_bytes.small_pool.allocated‚19569231268352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14100356509696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51808101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚895025743288138 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚910355966388736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122536316 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0'™N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚880949254637568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14076514078538 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260363947 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51808624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚765794840836096 trainer/global_step‚6599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896255636270592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚896255609879040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14103461916672 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122536842 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚896255636270592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚895028872185238 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260340809 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70728215 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884346525659473e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19569231268352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14100356509696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51808101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚895025743288138 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚910355966388736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122536316 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚910359098187264 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260340809 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚785364072104448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896255609879040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118258165 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19569178364416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142105776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 _runtime‚6630.921680972 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚910359098187264 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14100356509696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚765794812701696,KŽy2Ã- _wandbruntime‚6630  _runtime‚6630kúb\ÄÖ /loss/FRt3/assignment_loss‚3.467144412653787 1loss/FBt2/assignment_loss‚0.08093892798309962 0loss/SRqqt1/detection_loss‚0.960807111658407 epoch‚5 0loss/FRt4/assignment_loss‚3.7592538311367942 0loss/SRqqt2/detection_loss‚3.013340026140213 /loss/FBt3/detection_loss‚2.6282352524645187 _step‚542 0loss/FRt1/assignment_loss‚1.5733666548362144 1loss/FBt4/assignment_loss‚0.17447025228949542 1loss/SRqqt3/detection_loss‚1.8980601339839225 /loss/FRt1/detection_loss‚1.5265642535227997 $loss/total_loss‚44.0190692486946 trainer/global_step‚6599 % _timestamp‚1.7884346525678446e+09 1loss/FBt3/assignment_loss‚0.08763170861362614 .loss/FRt4/detection_loss‚3.240084381330581 0loss/SRqqt4/detection_loss‚2.259485363960266 /loss/FBt1/detection_loss‚0.5903418444526883 _runtime‚6630.922814411 /loss/FBt4/detection_loss‚2.0187243968248367 0loss/SRqqt2/assignment_loss‚1.60445269683133 2loss/SRqqt3/assignment_loss‚1.6683505722863037 2loss/FBt1/assignment_loss‚0.019818999933277473 /loss/FRt2/detection_loss‚2.6813116427510977 /loss/FBt2/detection_loss‚2.7175768486091068 0loss/FRt2/assignment_loss‚3.2129662819206715 2loss/SRqqt1/assignment_loss‚0.9102796286742172 2loss/SRqqt4/assignment_loss‚1.4762355100829154 /loss/FRt3/detection_loss‚2.4496285157544273O,-ÜaÅÛ /loss/FBt1/detection_loss‚0.5903418444526883 _runtime‚6630.922814411 /loss/FBt4/detection_loss‚2.0187243968248367 0loss/SRqqt2/assignment_loss‚1.60445269683133 2loss/SRqqt3/assignment_loss‚1.6683505722863037 2loss/FBt1/assignment_loss‚0.019818999933277473 /loss/FRt2/detection_loss‚2.6813116427510977 /loss/FBt2/detection_loss‚2.7175768486091068 0loss/FRt2/assignment_loss‚3.2129662819206715 2loss/SRqqt1/assignment_loss‚0.9102796286742172 2loss/SRqqt4/assignment_loss‚1.4762355100829154 /loss/FRt3/detection_loss‚2.4496285157544273 /loss/FRt3/assignment_loss‚3.467144412653787 1loss/FBt2/assignment_loss‚0.08093892798309962 0loss/SRqqt1/detection_loss‚0.960807111658407 epoch‚5 0loss/FRt4/assignment_loss‚3.7592538311367942 0loss/SRqqt2/detection_loss‚3.013340026140213 /loss/FBt3/detection_loss‚2.6282352524645187 _step‚542 0loss/FRt1/assignment_loss‚1.5733666548362144 1loss/FBt4/assignment_loss‚0.17447025228949542 1loss/SRqqt3/detection_loss‚1.8980601339839225 /loss/FRt1/detection_loss‚1.5265642535227997 $loss/total_loss‚44.0190692486946 trainer/global_step‚6599 % _timestamp‚1.7884346525678446e+09 1loss/FBt3/assignment_loss‚0.08763170861362614 .loss/FRt4/detection_loss‚3.240084381330581 0loss/SRqqt4/detection_loss‚2.259485363960266žNÖ<“Æ‚(:ˆ ê±åÔå©‹µ gpu.0.gpu‚64.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚292.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’=fÇ‚(:Û ê±åÔ”íð¶ disk.dm-0.in‚ 466.73828125 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.dm-1.in‚0.203125$ memory_percent‚99.57461357116699* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 disk.rbd1.out‚ 3.373046875 disk.dm-0.out‚57236.87109375 disk.drbd2265.out‚0 disk.sda2.in‚0 network.recv‚ 2.225684e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 466.73828125" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd2265.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 69.6953125$ disk./.usageGB‚282.4659080505371 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0% disk.nvme0c0n1.out‚57236.87109375 disk.sda3.in‚0.203125 disk.sda3.out‚ 250.34375 disk.sda2.out‚0! proc.memory.rssMB‚ 2959.453125 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 250.34375# disk.nvme0n1.out‚57236.87109375 network.sent‚ 1.6093452e+07 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 250.34375 cpu‚25.267354481271166) disk./.usagePercent‚18.956524093954522Ã]—žÈ‚(:“ ù±åÔ–¨”µ gpu.0.gpu‚74.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚140.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.167399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\¿ŠóvÉ‚(:ë ù±åÔÒãþ· network.recv‚ 2.233328e+06$ disk./.usageGB‚282.4910583496094 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.73828125 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.373046875' disk.nvme0c0n1.out‚57305.4697265625 disk.sda3.out‚ 250.94140625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 466.73828125$ memory_percent‚99.57699775695801 network.sent‚ 1.6202502e+07 disk.drbd2080.out‚0" disk.dm-0.out‚57305.4697265625* proc.memory.percent‚18.063068389892578 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0* disk./.usagePercent‚18.958211951627764 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 250.94140625 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 250.94140625& proc.memory.availableMB‚ 69.3046875 disk.dm-1.in‚0.203125 cpu‚25.266895555233802 proc.cpu.threads‚17 disk.sda3.in‚0.203125% disk.nvme0n1.out‚57305.4697265625! proc.memory.rssMB‚ 2959.453125xFt%“Ê‚(:ˆ ˆ²åÔ•¥¯µ gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚273.707' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gLzeË‚(:Ú ˆ²åÔ–¢Ê¸ disk.sda2.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 466.73828125 disk.drbd2080.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.drbd2265.in‚0' proc.memory.availableMB‚ 69.55078125* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 disk.nvme0n1.in‚ 466.73828125 disk.dm-1.out‚251.5! proc.memory.rssMB‚ 2959.453125 cpu‚25.266405245091843* disk./.usagePercent‚18.960349870546207 disk.sda.in‚0.203125 disk.drbd2265.out‚0 network.recv‚ 2.235206e+06$ disk./.usageGB‚282.5229148864746 disk.drbd2266.in‚0 disk.sda3.out‚251.5 disk.drbd1716.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚57360.70263671875# disk.dm-0.out‚57360.70263671875 disk.drbd2080.in‚0 disk.sda.out‚251.5$ memory_percent‚99.57549571990967 network.sent‚ 1.6205972e+07 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.73828125( disk.nvme0c0n1.out‚57360.70263671875 disk.sda1.out‚0 disk.sda2.in‚0öx–'2Ì- _wandbruntime‚6678  _runtime‚6678ölmìÉÍà _runtime‚6678.309706624 _step‚543 ' lr-AdamW/pg1‚0.00010229997933729464 ' lr-AdamW/pg2‚0.00010229997933729464 trainer/global_step‚6649 % _timestamp‚1.7884346999546056e+09âSÎÎÈ % _timestamp‚1.7884346999546056e+09 _runtime‚6678.309706624 _step‚543 ' lr-AdamW/pg1‚0.00010229997933729464 ' lr-AdamW/pg2‚0.00010229997933729464 trainer/global_step‚6649Ÿ‚m2Ï- _wandbruntime‚6678  _runtime‚6678§Ê+Ë(ЊP GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884346999555125e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902698846592000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14206979696128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14203874302464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14203874302464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚6678.310895237 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902698846592000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19713057971200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916905826288128 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚791015948000768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚771302945043968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143119406 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916902694502912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14179858531646 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143142512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚791016031149568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262224487 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143119406 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902698820200448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123410246 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71230615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚901469540800830 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚887289707697152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119105087 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916902694502912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123410765 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚887289682269184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14206979696128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52180150 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚901472669697722 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143142512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71230612 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚771302973178368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14182962000570 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52179634 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19713002956800h[ßÁÑP SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14179858531646 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143142512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚791016031149568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262224487 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143119406 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902698820200448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123410246 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71230615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚901469540800830 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚887289707697152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119105087 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916902694502912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123410765 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚887289682269184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14206979696128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52180150 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚901472669697722 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143142512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71230612 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚771302973178368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14182962000570 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52179634 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19713002956800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884346999555125e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902698846592000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14206979696128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14203874302464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14203874302464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚6678.310895237 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902698846592000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19713057971200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916905826288128 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚791015948000768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚771302945043968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143119406 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916902694502912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ­,œ™2Ò- _wandbruntime‚6679  _runtime‚6679æY'Ó”N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚887419079270400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902830314288640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71240868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71240865 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14185070793878 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143140496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚902830340680192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14209092023296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52187793 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262285999 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143140496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚901604175492246 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262262861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚887419104698368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _runtime‚6679.220867043 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚917039432703488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚771415384042496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119122365 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14205986616320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52187300 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902830340680192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123428661 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚901601046595146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119122365 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚771415355908096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123428165 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14205986616320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262285999 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19715942299136 trainer/global_step‚6649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262262861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119122371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143163628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚791131381342720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917036300904960 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119122371 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚791131298207232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143163628 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14181967324746 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123428165 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14205986616320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262285999 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19715942299136 trainer/global_step‚6649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262262861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119122371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143163628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚791131381342720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917036300904960 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119122371 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚791131298207232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143163628 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14181967324746 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143140496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚901604175492246 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262262861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚887419104698368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _runtime‚6679.220867043 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚917039432703488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚771415384042496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119122365 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14205986616320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52187300 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902830340680192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123428661 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¡Víöƒ2Õ- _wandbruntime‚6679  _runtime‚6679ߣœUUÖÏ 1loss/SRqqt3/assignment_loss‚1.409150424413383 /loss/FRt2/detection_loss‚2.9067777004735222 -loss/FRt3/detection_loss‚2.43851149559021 1loss/SRqqt3/detection_loss‚1.7670305124842203 .loss/FBt2/detection_loss‚2.324048182616631 2loss/SRqqt2/assignment_loss‚1.7220728132745315 /loss/FRt1/detection_loss‚1.5194642838826518 0loss/SRqqt1/detection_loss‚1.046423597126892 .loss/FBt4/detection_loss‚2.034153774380684 0loss/FRt3/assignment_loss‚2.7312928545475006 trainer/global_step‚6649 # _timestamp‚1.78843470086702e+09 _step‚546 1loss/FBt1/assignment_loss‚0.03443924089160758 epoch‚5 _runtime‚6679.222000393 0loss/FBt4/assignment_loss‚0.0861634502587269 .loss/FRt4/detection_loss‚4.457756693546589 &loss/total_loss‚45.403452654645434 /loss/FRt4/assignment_loss‚3.666901313341581 2loss/SRqqt1/assignment_loss‚0.9910556645967912 1loss/SRqqt4/assignment_loss‚1.924772673639758 0loss/FBt3/assignment_loss‚0.3868857025102303 1loss/SRqqt4/detection_loss‚2.5095931702646714 /loss/FRt2/assignment_loss‚3.687684988153392 0loss/FBt2/assignment_loss‚0.3622228534853396 1loss/SRqqt2/detection_loss‚2.7403141722596924 0loss/FRt1/assignment_loss‚1.5940360699170901 /loss/FBt1/detection_loss‚0.6205985907387609 /loss/FBt3/detection_loss‚2.4421024322509766Òý!7Z×Ô _step‚546 1loss/FBt1/assignment_loss‚0.03443924089160758 epoch‚5 _runtime‚6679.222000393 0loss/FBt4/assignment_loss‚0.0861634502587269 .loss/FRt4/detection_loss‚4.457756693546589 &loss/total_loss‚45.403452654645434 /loss/FRt4/assignment_loss‚3.666901313341581 2loss/SRqqt1/assignment_loss‚0.9910556645967912 1loss/SRqqt4/assignment_loss‚1.924772673639758 0loss/FBt3/assignment_loss‚0.3868857025102303 1loss/SRqqt4/detection_loss‚2.5095931702646714 /loss/FRt2/assignment_loss‚3.687684988153392 0loss/FBt2/assignment_loss‚0.3622228534853396 1loss/SRqqt2/detection_loss‚2.7403141722596924 0loss/FRt1/assignment_loss‚1.5940360699170901 /loss/FBt1/detection_loss‚0.6205985907387609 /loss/FBt3/detection_loss‚2.4421024322509766 1loss/SRqqt3/assignment_loss‚1.409150424413383 /loss/FRt2/detection_loss‚2.9067777004735222 -loss/FRt3/detection_loss‚2.43851149559021 1loss/SRqqt3/detection_loss‚1.7670305124842203 .loss/FBt2/detection_loss‚2.324048182616631 2loss/SRqqt2/assignment_loss‚1.7220728132745315 /loss/FRt1/detection_loss‚1.5194642838826518 0loss/SRqqt1/detection_loss‚1.046423597126892 .loss/FBt4/detection_loss‚2.034153774380684 0loss/FRt3/assignment_loss‚2.7312928545475006 trainer/global_step‚6649 # _timestamp‚1.78843470086702e+09¢Ò<¢žØ‚(:“ —²åÔ¤®¥µ gpu.0.gpu‚69.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚130.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.167599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”©>ÅÙ‚(:à —²åԣ滸 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 3.521484375 proc.cpu.threads‚17 network.sent‚ 1.6212038e+07 disk.dm-0.in‚ 466.73828125 disk.nvme0n1.in‚ 466.73828125 disk.sda2.out‚0* proc.memory.percent‚18.063068389892578) disk./.usagePercent‚18.96182037782288 disk.drbd2080.in‚0" disk.nvme0n1.out‚ 57403.9140625 disk.dm-1.out‚ 252.2734375 disk.drbd2266.out‚0$ memory_percent‚99.57873821258545 cpu‚25.265922921478797 network.recv‚ 2.239817e+06% disk./.usageGB‚282.54482650756836$ disk.nvme0c0n1.out‚ 57403.9140625 disk.sda1.in‚0 disk.sda.out‚ 252.2734375 disk.drbd2265.out‚0' proc.memory.availableMB‚ 69.01953125" disk.nvme0c0n1.in‚ 466.73828125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.rbd1.in‚0 disk.dm-0.out‚ 57403.9140625}!o\¦ disk.sda3.in‚0.203125 disk.sda3.out‚ 252.2734375 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2959.453125 disk.drbd2080.out‚0 disk.drbd2265.in‚0Pó¢“Ú‚(:ˆ ¦²åÔ«‘µ gpu.0.gpu‚64.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚270.003' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èbQYnÛ‚(:ã ¦²åÔ–†ú· disk.sda.out‚ 252.609375 disk.dm-1.in‚0.203125 disk.sda3.out‚ 252.609375% proc.memory.availableMB‚ 69.484375 proc.cpu.threads‚17 network.recv‚ 2.245807e+06 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.rbd1.out‚ 3.521484375" disk.nvme0c0n1.in‚ 466.73828125 disk.dm-1.out‚ 252.609375' disk.nvme0c0n1.out‚57452.5439453125 disk.rbd1.in‚0 network.sent‚ 1.632073e+07 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.73828125 disk.sda2.out‚0! proc.memory.rssMB‚ 2959.453125 cpu‚25.265445777099217" disk.dm-0.out‚57452.5439453125 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0* disk./.usagePercent‚18.962937338694775 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚57452.5439453125$ memory_percent‚99.57590103149414 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.73828125 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.063068389892578$ disk./.usageGB‚282.5614700317383LÔ⛓Ü‚(:ˆ µ²åÔÚ£›µ gpu.0.gpu‚84.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÇ„pyÝ‚(:î µ²åÔÅšÕ¶ disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.73828125( disk.nvme0c0n1.out‚57532.62646484375 disk.drbd1716.in‚0# memory_percent‚99.5743989944458* proc.memory.percent‚18.063068389892578 disk.drbd2265.out‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 cpu‚25.264939580964807# disk./.usageGB‚282.589298248291 disk.rbd0.in‚0 disk.sda.out‚ 253.40234375 network.recv‚ 2.250665e+06 disk.rbd1.out‚ 3.5537109375# disk.dm-0.out‚57532.62646484375 network.sent‚ 1.6326279e+07* disk./.usagePercent‚18.964804913657055 disk.sda2.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚57532.62646484375 disk.drbd1716.out‚0 disk.sda.in‚0.203125' proc.memory.availableMB‚ 69.73046875 disk.nvme0n1.in‚ 466.73828125 disk.sda2.in‚0 disk.dm-1.out‚ 253.40234375 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2959.453125 proc.cpu.threads‚17 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.73828125 disk.sda3.out‚ 253.40234375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0çb2Þ- _wandbruntime‚6726  _runtime‚6726ÕnAÉßà _runtime‚6726.727484949 _step‚547 ' lr-AdamW/pg1‚0.00010082342550488548 ' lr-AdamW/pg2‚0.00010082342550488548 trainer/global_step‚6699 % _timestamp‚1.7884347483723836e+096uVœÎàÈ % _timestamp‚1.7884347483723836e+09 _runtime‚6726.727484949 _step‚547 ' lr-AdamW/pg1‚0.00010082342550488548 ' lr-AdamW/pg2‚0.00010082342550488548 trainer/global_step‚6699£8)*«2á- _wandbruntime‚6726  _runtime‚6726¼‹ÜÇ(âŠP SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144177258 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144177258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52561003 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144200364 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚923583029019136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71743262 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14288415246778 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚923586160804352 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 264146539 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _runtime‚6726.728741581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119969287 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144200364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚908047973004730 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71743265 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚909273551001600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19859787694080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Vì‚(ãP =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚908047973004730 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71743265 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚909273551001600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19859787694080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144177258 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144177258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52561003 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144200364 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚923583029019136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71743262 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14288415246778 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚923586160804352 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 264146539 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _runtime‚6726.728741581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119969287 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144200364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552¤t»áf2ä- _wandbruntime‚6727  _runtime‚6727'ó£â  å’N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚923719767219712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚796898705808896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚908179478799254 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14290524040086 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264208051 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264184913 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144198348 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14311616722944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 $ _timestamp‚1.788434749282249e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚923719767219712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚777035899114496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264208051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚908176349902154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 trainer/global_step‚6699 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚909405018698240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚893888954759168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119986571 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893888929331200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14314722129920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14311616722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71753515 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚777035927248896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 FDeviceStatsMonitor.on_train_batch_end/active_bytZU½xes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71753518 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52568682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14287420570954 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124322200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19862721461760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124321673 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264208051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚908176349902154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 trainer/global_step‚6699 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚909405018698240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚893888954759168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119986571 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893888929331200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14314722129920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14311616722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71753515 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚777035927248896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71753518 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52568682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14287420570954 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124322200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19862721461760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124321673 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125199108 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120833481 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914620147414846 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚930266495320576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚915848229019648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145235110 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14418239909376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚930266495320576 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14390765024062 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚930263363535360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72255915 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266068591 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚782544059591168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20006613594624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145235110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6774.726875039 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145258216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52943770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120833487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚915848255411200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266091703 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚782544031456768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 trainer/global_step‚6749 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14415134515712 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125199685 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 % _timestamp‚1.7884347963715947e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Gðký õP % _timestamp‚1.7884347963715947e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚914623276311738 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14393868492986 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125199108 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120833481 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914620147414846 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚930266495320576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚915848229019648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145235110 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14418239909376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚930266495320576 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14390765024062 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚930263363535360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72255915 GADeviceStatsMonitor.on_train_batch_start/oversize_allvª-€ocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266068591 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚782544059591168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20006613594624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145235110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6774.726875039 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145258216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52943770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120833487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚915848255411200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266091703 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚782544031456768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 trainer/global_step‚6749 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14415134515712 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125199685 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456¨®[ ï2ö- _wandbruntime‚6775  _runtime‚67756åBÚ'÷”N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266130103 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _step‚553 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚930400101735936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14420352236544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚914751653209162 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14417246829568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125217309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120850771 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚900358779392000 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72266165 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6775.653533311 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266130103 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚915979749499392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145279332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52950596 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120850765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120850765 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚802665906853376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚782656442320896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚930400101735936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884347972978213e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120850771 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266106965 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145256200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚782656470455296 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20009464532480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145279332 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚930396969937408 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125216761 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878JY~¢'ø™N I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145256200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚782656470455296 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20009464532480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145279332 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚930396969937408 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125216761 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266130103 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _step‚553 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚930400101735936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14420352236544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚914751653209162 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14417246829568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125217309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120850771 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚900358779392000 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72266165 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6775.653533311 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266130103 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚915979749499392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145279332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52950596 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120850765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120850765 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚802665906853376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚782656442320896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚930400101735936 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884347972978213e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120850771 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266106965 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 P\ûÖ /loss/FBt3/detection_loss‚2.1017628146542444 /loss/FBt4/detection_loss‚1.7384508322266972 $loss/total_loss‚43.2216540252598 0loss/FRt1/assignment_loss‚1.7486111877426025 1loss/SRqqt3/detection_loss‚2.2288912200566493 .loss/FBt2/detection_loss‚2.698271998337337 2loss/SRqqt2/assignment_loss‚1.7105420251687369 1loss/SRqqt3/assignment_loss‚1.076422815521558 0loss/FBt2/assignment_loss‚0.4062462249005497 .loss/FRt2/detection_loss‚2.462880476927146 1loss/SRqqt2/detection_loss‚2.4056978772083917 2loss/SRqqt1/assignment_loss‚1.1366573098736505 0loss/FBt3/assignment_loss‚0.2015890952768539 1loss/SRqqt4/detection_loss‚2.7930271097204904 2loss/SRqqt4/assignment_loss‚1.8027070930058307 2loss/FBt1/assignment_loss‚0.004240266763148688 /loss/FRt1/detection_loss‚1.7280008982746833 .loss/FRt3/detection_loss‚2.301913207769394 /loss/FRt4/detection_loss‚2.8502553367614745 .loss/FRt2/assignment_loss‚3.58460441002479 1loss/FBt4/assignment_loss‚0.15939268314441707 .loss/FBt1/detection_loss‚0.773174127988648 epoch‚5 _step‚554 /loss/FRt4/assignment_loss‚3.451717336177826 0loss/SRqqt1/detection_loss‚1.042186934617348 $ _timestamp‚1.788434797299635e+09 0loss/FRt3/assignment_loss‚2.8144107431173326 trainer/global_step‚6749 _runtime‚6775.654652694ªP*ò"“ü‚(:ˆ ñ²åÔåΗµ gpu.0.gpu‚81.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚290.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×¹ótý‚(:é ñ²åÔ²ì÷¶ disk.sda1.out‚0! proc.memory.rssMB‚ 2959.453125 cpu‚25.26307930706876* disk./.usagePercent‚18.969578174132966 disk.nvme0n1.in‚ 466.73828125 disk.dm-0.in‚ 466.73828125 disk.sda2.in‚0 disk.dm-1.out‚ 255.3359375 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚57840.32666015625 network.sent‚ 1.6448153e+07 network.recv‚ 2.266132e+06 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.sda3.out‚ 255.3359375$ memory_percent‚99.57542419433594# proc.memory.availableMB‚69.5625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.063068389892578 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.61669921875$ disk./.usageGB‚282.6604232788086 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 255.3359375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚57840.32666015625 disk.dm-1.in‚0.203125# disk.dm-0.out‚57840.32666015625 disk.sda3.in‚0.203125bY㺓þ‚(:ˆ €³åÔÖЪµ gpu.0.gpu‚71.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚223.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.7678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kFÒ±ÿ‚(:× €³åÔÄ· disk.drbd2266.out‚0 cpu‚25.26260925733308 disk.drbd2265.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚18.972060167177425" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd2080.in‚0 disk.sda.out‚255.75* proc.memory.percent‚18.063068389892578 disk.dm-1.in‚0.203125 disk.sda3.out‚255.75 disk.drbd2265.in‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.7382xT½±8125 disk.rbd1.in‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚57939.26025390625 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2959.453125 network.recv‚ 2.270944e+06 disk.dm-0.in‚ 466.73828125 disk.sda2.in‚0 disk.dm-1.out‚255.75 disk.rbd1.out‚ 3.6796875$ disk./.usageGB‚282.6974067687988# disk.dm-0.out‚57939.26025390625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0# memory_percent‚99.5762825012207% proc.memory.availableMB‚ 69.421875 network.sent‚ 1.6555185e+07 disk.drbd2080.out‚0& disk.nvme0n1.out‚57939.26025390625 disk.drbd2266.in‚0u¹˜éž€‚(:“ ³åÔÑžùµ gpu.0.gpu‚86.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.839200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v/8Êt‚(:é ³åÔÛÿݶ disk.sda3.out‚ 256.5859375' proc.memory.availableMB‚ 68.99609375! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578 network.sent‚ 1.6559652e+07 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 466.73828125 proc.cpu.threads‚17) disk./.usagePercent‚18.97332586842671 disk.nvme0n1.in‚ 466.73828125" disk.nvme0c0n1.in‚ 466.73828125# disk.dm-0.out‚58034.79052734375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0$ memory_percent‚99.57888126373291 network.recv‚ 2.274733e+06& disk.nvme0n1.out‚58034.79052734375 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 256.5859375 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda2.in‚0 disk.drbd2266.in‚0 cpu‚25.262145703997447$ disk./.usageGB‚282.7162666320801 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 256.5859375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚58034.79052734375 disk.sda3.in‚0.203125IÕ%Î2‚- _wandbruntime‚6823  _runtime‚6823ÁKõƃÀ _step‚555 & lr-AdamW/pg1‚9.787153939134478e-05 & lr-AdamW/pg2‚9.787153939134478e-05 trainer/global_step‚6799 $ _timestamp‚1.788434844708492e+09 _runtime‚6823.063549544iQ~Ë„Å _runtime‚6823.063549544 _step‚555 & lr-AdamW/pg1‚9.787153939134478e-05 & lr-AdamW/pg2‚9.787153939134478e-05 trainer/global_step‚6799 $ _timestamp‚1.788434844708492e+09«²OÔ2…- _wandbruntime‚6823  _runtime‚6823N¼G–(†ŠP QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146316068 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚922422959820800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126092211 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚906699257879552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚922422933429248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808317643118080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14523870016000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146316068 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14496218270270 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126092744 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72768565 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146292962 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚6799 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚936943698051584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚906699232451584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚808317726266880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚936946829836800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121697687 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20153068454912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚6823.064869155 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚936943698051584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14499321739194 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146292962 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268013755 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121697681 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚936946829836800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72768562 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20153123469312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121697687 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 % _timestamp‚1.7884348447094374e+09-4­(‡P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14523870016000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146316068 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14496218270270 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126092744 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72768565 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146292962 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚6799 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚936943698051584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚906699232451584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚808317726266880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚936946829836800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121697687 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20153068454912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚6823.064869155 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚936943698051584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14499321739194 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146292962 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268013755 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121697681 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚936946829836800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72768562 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20153123469312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121697687 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 % _timestamp‚1.7884348447094374e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146316068 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚922422959820800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126092211 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚906699257879552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚922422933429248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808317643118080¬0àª2ˆ- _wandbruntime‚6823  _runtime‚6823¾\ŠÅ¤$‰”N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚6823.981406281 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268029017 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14522876936192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146337184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268052155 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884348456258545e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72778818 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚921326956516170 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20155994335232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937077304453632 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14525982343168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚808432979862528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20156049336320 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146314052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126110146 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚906828629452800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268052155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14498327063370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚921330085413270 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚808433062998016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14501430532502 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚922554427517440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚906828654880768 trainer/global_step‚6799 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121714965 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121714971 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53331331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocatMQwved‚53331870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚788277013661696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088åT*:'Š™N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚922554427517440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚906828654880768 trainer/global_step‚6799 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121714965 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121714971 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53331331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53331870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚788277013661696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚6823.981406281 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268029017 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14522876936192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146337184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268052155 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884348456258545e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72778818 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚921326956516170 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20155994335232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937077304453632 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14525982343168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚808432979862528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20156049336320 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146314052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126110146 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚906828629452800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268052155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14498327063370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚921330085413270 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚808433062998016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14501430532502 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440­D—šà2‹- _wandbruntime‚6823  _runtime‚6823zpŽXŒÒ /loss/FRt3/assignment_loss‚3.120946536804068 2loss/SRqqt1/assignment_loss‚0.9346489396730536 /loss/FRt2/detection_loss‚2.2160468995571136 /loss/FRt4/detection_loss‚3.2962401087085405 /loss/FBt1/detection_loss‚0.7880754972759046 epoch‚5 0loss/FRt2/assignment_loss‚3.5649346709251404 1loss/SRqqt1/detection_loss‚0.9222126000021633 _runtime‚ 6823.98248694 1loss/FBt2/assignment_loss‚0.46702329206746074 0loss/SRqqt3/detection_loss‚2.499874551748407 .loss/FBt2/detection_loss‚2.276640224456787 _step‚558 0loss/SRqqt4/detection_loss‚2.955247563123703 .loss/FBt3/detection_loss‚2.597332375390189 &loss/total_loss‚43.938994568904306 trainer/global_step‚6799 -loss/FRt1/assignment_loss‚1.5279897839304 1loss/SRqqt2/assignment_loss‚1.398411393813465 3loss/FBt1/assignment_loss‚0.0018266597062259592 1loss/FBt3/assignment_loss‚0.20945971658719284 1loss/FBt4/assignment_loss‚0.09654036376731716 0loss/SRqqt2/detection_loss‚2.564870463765186 " _timestamp‚1.7884348456275e+09 2loss/SRqqt4/assignment_loss‚1.4954694915562867 /loss/FBt4/detection_loss‚1.5934630153434617 0loss/FRt4/assignment_loss‚3.6890300462643304 2loss/SRqqt3/assignment_loss‚1.1923165616804157 /loss/FRt1/detection_loss‚1.5811493279382822 /loss/FRt3/detection_loss‚2.9492444848192148aÖTm]× 0loss/SRqqt4/detection_loss‚2.955247563123703 .loss/FBt3/detection_loss‚2.597332375390189 &loss/total_loss‚43.938994568904306 trainer/global_step‚6799 -loss/FRt1/assignment_loss‚1.5279897839304 1loss/SRqqt2/assignment_loss‚1.398411393813465 3loss/FBt1/assignment_loss‚0.0018266597062259592 1loss/FBt3/assignment_loss‚0.20945971658719284 1loss/FBt4/assignment_loss‚0.09654036376731716 0loss/SRqqt2/detection_loss‚2.564870463765186 " _timestamp‚1.7884348456275e+09 2loss/SRqqt4/assignment_loss‚1.4954694915562867 /loss/FBt4/detection_loss‚1.5934630153434617 0loss/FRt4/assignment_loss‚3.6890300462643304 2loss/SRqqt3/assignment_loss‚1.1923165616804157 /loss/FRt1/detection_loss‚1.5811493279382822 /loss/FRt3/detection_loss‚2.9492444848192148 /loss/FRt3/assignment_loss‚3.120946536804068 2loss/SRqqt1/assignment_loss‚0.9346489396730536 /loss/FRt2/detection_loss‚2.2160468995571136 /loss/FRt4/detection_loss‚3.2962401087085405 /loss/FBt1/detection_loss‚0.7880754972759046 epoch‚5 0loss/FRt2/assignment_loss‚3.5649346709251404 1loss/SRqqt1/detection_loss‚0.9222126000021633 _runtime‚ 6823.98248694 1loss/FBt2/assignment_loss‚0.46702329206746074 0loss/SRqqt3/detection_loss‚2.499874551748407 .loss/FBt2/detection_loss‚2.276640224456787 _step‚558®–"‰àjŽ‚(:ß ž³åÔö့ disk.drbd2080.in‚0 disk.sda.in‚0.203125% disk.nvme0n1.out‚58112.0693359375 disk.sda2.in‚0' disk.nvme0c0n1.out‚58112.0693359375 network.recv‚ 2.277483e+06$ disk./.usageGB‚282.7439384460449 disk.dm-1.out‚ 256.96875 cpu‚25.261679138682258 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 466.73828125$ memory_percent‚99.57985877990723 disk.drbd1716.out‚0 disk.sda.out‚ 256.96875 disk.nvme0n1.in‚ 466.73828125" disk.dm-0.out‚58112.0693359375& proc.memory.availableMB‚ 68.8359375* proc.memory.percent‚18.063068389892578 disk.dm-0.in‚ 466.73828125 disk.sda3.in‚0.203125 disk.sda3.out‚ 256.96875! proc.memory.rssMB‚ 2959.453125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda1.out‚0 network.sent‚ 1.6564368e+07) disk./.usagePercent‚18.97518294708009 disk.drbd1716.in‚0 disk.dm-1.in‚0.203125Q?!ýž‚(:“ ž³åÔ•‰Î· gpu.0.gpu‚70.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚127.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.564799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òdx“‚(:ˆ ­³åÔã÷‚¶ gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚279.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.9812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š³uµr‘‚(:ç ­³åÔ‹¹ð¶ disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 257.59765625 cpu‚25.26121763871628) disk./.usagePercent‚18.97664116599512 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875' disk.nvme0c0n1.out‚58182.0068359375 disk.sda.in‚0.203125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 466.73828125" disk.dm-0.out‚58182.0068359375 disk.sda3.in‚0.203125 disk.sda3.out‚ 257.59765625 disk.rbd0.in‚0 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2959.453125 network.recv‚ 2.283696e+06 disk.drbd2080.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚58182.0068359375 disk.dm-1.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.73828125 disk.nvme0n1.in‚ 466.73828125* proc.memory.percent‚18.063068389892578 network.sent‚ 1.6671739e+07$ disk./.usageGB‚282.7656669616699 disk.drbd2265.in‚0& proc.memory.availableMB‚ 68.5703125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 257.59765625$ memory_percent‚99.58148002624512ÚVçüž’‚(:“ ¼³åÔªƒú¶ gpu.0.gpu‚68.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!'¢s“‚(:è ¼³åÔ—厹! proc.memory.rssMB‚ 2959.453125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.dm-0.in‚ 466.73828125 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 257.99609375' disk.nvme0c0n1.out‚58250.9052734375 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda3.in‚0.203125' proc.memory.availableMB‚ 69.06640625 disk.nvme0n1.in‚ 466.73828125% disk.nvme0n1.out‚58250.9052734375 disk.sda3.out‚ 257.99609375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 257.99609375 network.sent‚ 1.6677182e+07 network.recv‚ 2.287102e+06 disk.sda2.out‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.063068389892578$ disk./.usageGB‚282.7929878234863 disk.sda2.in‚0" disk.dm-0.out‚58250.9052734375 disk.sda1.in‚0 disk.sda.in‚0.203125 cpu‚25.26077643826823 proc.cpu.threads‚17 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.sda1.out‚0$ memory_percent‚99.57845211029053) disk./.usagePercent‚18.97847469195535 disk.drbd1716.in‚0µ?K_“”‚(:ˆ ˳åÔø¾‡µ gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@O{q•‚(:æ ˳åÔʮǶ proc.cpu.threads‚17) disk./.usagePercent‚18.98027749701147% disk./.usageGB‚282.81985092163086 disk.sda1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.58300590515137 cpu‚25.260337485951577 disk.sda2.out‚0 disk.rbd1.out‚ 3.6796875' disk.nvme0c0n1.out‚58322.8291015625 disk.sda1.out‚0 disk.sda.out‚ 258.6953125 disk.nvme0n1.in‚ 466.73828125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.73828125* proc.memory.percent‚18.063068389892578 network.recv‚ 2.291399e+06 disk.drbd2265.in‚0% disk.nvme0n1.out‚58322.8291015625 disk.drbd2266.out‚0 network.sent‚ 1.6683242e+07 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2266.in‚0 disk.sda3.out‚ 258.6953125 disk.rbd0.out‚0 disk.dm-1.out‚ 258.6953125 disk.dm-0.in‚ 466.73828125 disk.sda2.in‚0" disk.dm-0.out‚58322.8291015625 disk.sda3.in‚0.203125& proc.memory.availableMB‚ 68.3203125 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125! proc.memory.rssMB‚ 2959.453125{ÛýÒ2–- _wandbruntime‚6871  _runtime‚6871 ºªÅ—¿ trainer/global_step‚6849 % _timestamp‚1.7884348930268657e+09 _runtime‚6871.381913064 _step‚559 % lr-AdamW/pg1‚9.63968222788549e-05 % lr-AdamW/pg2‚9.63968222788549e-05^44Ê˜Ä trainer/global_step‚6849 % _timestamp‚1.7884348930268657e+09 _runtime‚6871.381913064 _step‚559 % lr-AdamW/pg1‚9.63968222788549e-05 % lr-AdamW/pg2‚9.63968222788549e-05¯žwpi2™- _wandbruntime‚6871  _runtime‚6871æ6(šŠP SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚943624032567808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14626394728960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20299905523712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884348930276363e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269912695 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147350814 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147350814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚927770754028862 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73281212 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14629500122624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122561887 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚943627164353024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚928997637838848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚814084968378880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53705494 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122561881 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚913169107940352 _runtime‚6871.382908082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚928997664230400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚943627164353024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚793785146003968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147373920 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984êˆïTÚ›P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚927773882925754 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚928997637838848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147373920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14626394728960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20299905523712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884348930276363e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269912695 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147350814 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147350814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚927770754028862 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73281212 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14629500122624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122561887 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚943627164353024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚928997637838848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚814084968378880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53705494 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122561881 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚913169107940352 _runtime‚6871.382908082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚928997664230400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚943627164353024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚793785146003968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147373920 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚943624032567808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127004570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚943760770768384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚929129131927040 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73291465 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122579171 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14631612449792 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚929129158318592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14628507042816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚943760770768384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269974207 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6872.287746124 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122579171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147371904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚913298479513600 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147395036 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20302843670016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269951069 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127004570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚943760770768384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚929129131927040 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73291465 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122579171 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14631612449792 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚929129158318592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14628507042816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚943760770768384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269974207 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6872.287746124 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122579171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147371904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚913298479513600 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147395036 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20302843670016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0±…âÊ2Ÿ- _wandbruntime‚6872  _runtime‚6872N¹~Y Ó 3loss/FBt1/assignment_loss‚0.0028990515716202257 /loss/FRt1/detection_loss‚1.4554135314588035 /loss/FRt2/detection_loss‚2.3897246373327157 1loss/SRqqt1/detection_loss‚0.7818111604201081 0loss/SRqqt3/detection_loss‚1.973667358027564 0loss/FRt3/assignment_loss‚3.2773819219681526 /loss/FRt4/assignment_loss‚3.176727819442749 epoch‚5 _step‚562 /loss/FRt1/assignment_loss‚1.347228582416262 /loss/FRt4/detection_loss‚3.1535393047332763 .loss/FBt4/detection_loss‚2.434847821082388 _runtime‚6872.288925801 1loss/FBt2/assignment_loss‚0.12881746683963094 0loss/FBt4/assignment_loss‚0.2619288930784803 /loss/FBt2/detection_loss‚1.8799740777296179 trainer/global_step‚6849 % _timestamp‚1.7884348939339595e+09 2loss/SRqqt1/assignment_loss‚1.0171381494880694 .loss/FRt3/detection_loss‚2.967944065409322 /loss/FRt2/assignment_loss‚3.770410478115082 1loss/SRqqt3/assignment_loss‚1.642531095776293 1loss/SRqqt4/assignment_loss‚2.080967093152659 1loss/FBt3/assignment_loss‚0.17811772228450168 0loss/SRqqt4/detection_loss‚2.437773721558707 /loss/FBt3/detection_loss‚2.6474309384822847 %loss/total_loss‚44.11455414648113 1loss/SRqqt2/assignment_loss‚1.794411438314811 0loss/SRqqt2/detection_loss‚2.605039521403935 /loss/FBt1/detection_loss‚0.7088282963941022üDV^¡Ø /loss/FRt1/detection_loss‚1.4554135314588035 /loss/FRt2/detection_loss‚2.3897246373327157 1loss/SRqqt1/detection_loss‚0.7818111604201081 0loss/SRqqt3/detection_loss‚1.973667358027564 0loss/FRt3/assignment_loss‚3.2773819219681526 /loss/FRt4/assignment_loss‚3.176727819442749 epoch‚5 _step‚562 /loss/FRt1/assignment_loss‚1.347228582416262 /loss/FRt4/detection_loss‚3.1535393047332763 .loss/FBt4/detection_loss‚2.434847821082388 _runtime‚6872.288925801 1loss/FBt2/assignment_loss‚0.12881746683963094 0loss/FBt4/assignment_loss‚0.2619288930784803 /loss/FBt2/detection_loss‚1.8799740777296179 trainer/global_step‚6849 % _timestamp‚1.7884348939339595e+09 2loss/SRqqt1/assignment_loss‚1.0171381494880694 .loss/FRt3/detection_loss‚2.967944065409322 /loss/FRt2/assignment_loss‚3.770410478115082 1loss/SRqqt3/assignment_loss‚1.642531095776293 1loss/SRqqt4/assignment_loss‚2.080967093152659 1loss/FBt3/assignment_loss‚0.17811772228450168 0loss/SRqqt4/detection_loss‚2.437773721558707 /loss/FBt3/detection_loss‚2.6474309384822847 %loss/total_loss‚44.11455414648113 1loss/SRqqt2/assignment_loss‚1.794411438314811 0loss/SRqqt2/detection_loss‚2.605039521403935 /loss/FBt1/detection_loss‚0.7088282963941022 3loss/FBt1/assignment_loss‚0.0028990515716202257²Öœ5“¢‚(:ˆ Ú³åÔ–Àµ gpu.0.gpu‚43.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚307.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.5912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í…ºqi£‚(:Þ Ú³åÔß…Ž·% disk.nvme0n1.out‚58388.9931640625 disk.sda.in‚0.203125 disk.rbd0.out‚0' proc.memory.availableMB‚ 68.76171875 disk.drbd2080.in‚0 disk.sda.out‚259.0625 disk.dm-1.out‚259.0625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875' disk.nvme0c0n1.out‚58388.9931640625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚18.063068389892578 network.sent‚ 1.6688903e+07* disk./.usagePercent‚18.982361142332653 disk.nvme0n1.in‚ 466.73828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125" disk.dm-0.out‚58388.9931640625 disk.drbd1716.in‚0$ memory_percent‚99.58031177520752 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.73828125 proc.cpu.threads‚17 network.recv‚ 2.294806e+06$ disk./.usageGB‚282.8508987426758 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2959.453125" disk.nvme0c0n1.in‚ 466.73828125 disk.sda3.out‚259.0625 cpu‚25.259840520974997¯Žª1¤‚(:“ é³åÔ±ªµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚282.624' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.524800000000006 gpu.0.smCloZFn(mck‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E‡5d¥‚(:Ù é³åÔɯô· disk.rbd0.in‚0% disk.nvme0c0n1.out‚58469.96484375% proc.memory.availableMB‚ 68.515625* proc.memory.percent‚18.063068389892578 disk.sda3.out‚ 259.65625! proc.memory.rssMB‚ 2959.453125" disk.nvme0c0n1.in‚ 466.73828125$ memory_percent‚99.58181381225586 disk.dm-0.out‚58469.96484375 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd1.in‚0 cpu‚25.259380047143658 proc.cpu.threads‚17 network.recv‚ 2.300861e+06 disk.dm-0.in‚ 466.73828125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.6796875$ disk./.usageGB‚282.8726921081543 disk.sda1.in‚0 disk.sda.out‚ 259.65625 disk.nvme0n1.in‚ 466.73828125 disk.sda2.out‚0 disk.dm-1.out‚ 259.65625 disk.drbd2266.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚18.983823713375767 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2265.in‚0# disk.nvme0n1.out‚58469.96484375 network.sent‚ 1.6796154e+07 õ—¦‚(:’ ø³åԞؽµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.438' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.88759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+¦±ad§‚(:Ù ø³åÔüõ¶* proc.memory.percent‚18.063068389892578 network.recv‚ 2.303101e+06 disk.nvme0n1.in‚ 466.73828125 disk.sda3.out‚ 260.51953125 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 58517.390625 disk.rbd0.out‚0$ memory_percent‚99.58229064941406 cpu‚25.25893495353965 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda2.in‚0 disk.dm-1.out‚ 260.51953125 disk.drbd2266.out‚0# proc.memory.availableMB‚68.4375 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 network.sent‚ 1.6799755e+07% disk./.usageGB‚282.89521408081055 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2959.453125! disk.nvme0n1.out‚ 58517.390625 disk.dm-0.in‚ 466.73828125 disk.sda.out‚ 260.51953125) disk./.usagePercent‚18.98533518185792# disk.nvme0c0n1.out‚ 58517.390625 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0|ȱ2¨- _wandbruntime‚6919  _runtime‚6919BúÈ/Ç©Á _step‚563 & lr-AdamW/pg1‚9.492333229822843e-05 & lr-AdamW/pg2‚9.492333229822843e-05 trainer/global_step‚6899 % _timestamp‚1.7884349408909984e+09 _runtime‚6919.246099067ðo»ÌªÆ & lr-AdamW/pg1‚9.492333229822843e-05 & lr-AdamW/pg2‚9.492333229822843e-05 trainer/global_step‚6899 % _timestamp‚1.7884349408909984e+09 _runtime‚6919.246099067 _step‚563³¿û2«- _wandbruntime‚6919  _runtime‚6919ê#ö(¬ŠP NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799405689210368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14735130229248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚950304367084032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚935572368640000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14732024835584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271857859 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚950307498869248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14735130229248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20446413154816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚919638932573184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123426087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚819852019216384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148431772 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73793865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚950304367084032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123426087 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚819852102365184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123426081 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127879119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54085254 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123426081 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚935572342248448 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148408666 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚935572342248448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20446358140416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934346057335870 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚934349186232762 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148408666 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚950307498869248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚919638958001152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14732024835584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148408666 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚950307498869248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚919638958001152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14732024835584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14735130229248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚950304367084032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚935572368640000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14732024835584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271857859 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚950307498869248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14735130229248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20446413154816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚919638932573184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123426087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚819852019216384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148431772 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73793865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚950304367084032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456´ä¢uõ2®- _wandbruntime‚6920  _runtime‚6920üÿ/ÅŒ"¯”N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚6899 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123443365 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799518100074496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123443371 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148429756 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14712337024918 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14734137149440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271896259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123443365 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 % _timestamp‚1.7884349418012712e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127896375 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54092787 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _step‚565 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123443371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚950437973486080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20449342429184 _runtime‚6920.157200442 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271896259 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚919768355002368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73804118 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚799518071940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚935703862728192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73804115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚934480692027286 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14737242556416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚935703836336640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148452888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚819967442503680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20449285330944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271896259 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚919768355002368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73804118 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚799518071940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚935703862728192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73804115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚934480692027286 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14737242556416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚935703836336640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148452888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚819967442503680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20449285330944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271896259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123443365 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 % _timestamp‚1.7884349418012712e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127896375 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54092787 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624µ|ÒGÖ2±- _wandbruntime‚6920  _runtime‚6920:³YZ²Ô _runtime‚6920.158609071 1loss/SRqqt4/assignment_loss‚1.798733411622899 .loss/FRt3/detection_loss‚2.430907671650251 1loss/SRqqt2/detection_loss‚2.3144371112187705 .loss/FBt1/detection_loss‚0.608908352183385 2loss/SRqqt2/assignment_loss‚1.5609213688382597 2loss/SRqqt3/assignment_loss‚1.9618426529535402 1loss/SRqqt4/detection_loss‚2.4171601895775114 _step‚566 /loss/FRt3/assignment_loss‚3.243855007820659 1loss/FBt3/assignment_loss‚0.12719827231485326 1loss/SRqqt1/detection_loss‚0.9611090969463641 2loss/FBt1/assignment_loss‚0.004605334372187069 /loss/FRt1/detection_loss‚1.5323038434698468 /loss/FBt2/detection_loss‚2.3517847110827765 trainer/global_step‚6899 /loss/FRt2/detection_loss‚2.3498188039151633 0loss/FBt2/assignment_loss‚0.4601587503918078 0loss/FBt4/assignment_loss‚0.1470227425340259 .loss/FBt4/detection_loss‚2.281996035150119 $loss/total_loss‚43.0230985240008 0loss/FRt1/assignment_loss‚1.9253669693356468 0loss/FRt2/assignment_loss‚3.4197955654888617 0loss/FRt4/assignment_loss‚3.2502317375996532 2loss/SRqqt1/assignment_loss‚0.8854945223270492 /loss/FRt4/detection_loss‚2.6756508034818314 1loss/SRqqt3/detection_loss‚1.7555805863812566 .loss/FBt3/detection_loss‚2.558214983344078 $ _timestamp‚1.788434941803598e+09 epoch‚5¶¡9_³Ù $loss/total_loss‚43.0230985240008 0loss/FRt1/assignment_loss‚1.9253669693356468 0loss/FRt2/assignment_loss‚3.4197955654888617 0loss/FRt4/assignment_loss‚3.2502317375996532 2loss/SRqqt1/assignment_loss‚0.8854945223270492 /loss/FRt4/detection_loss‚2.6756508034818314 1loss/SRqqt3/detection_loss‚1.7555805863812566 .loss/FBt3/detection_loss‚2.558214983344078 $ _timestamp‚1.788434941803598e+09 epoch‚5 _runtime‚6920.158609071 1loss/SRqqt4/assignment_loss‚1.798733411622899 .loss/FRt3/detection_loss‚2.430907671650251 1loss/SRqqt2/detection_loss‚2.3144371112187705 .loss/FBt1/detection_loss‚0.608908352183385 2loss/SRqqt2/assignment_loss‚1.5609213688382597 2loss/SRqqt3/assignment_loss‚1.9618426529535402 1loss/SRqqt4/detection_loss‚2.4171601895775114 _step‚566 /loss/FRt3/assignment_loss‚3.243855007820659 1loss/FBt3/assignment_loss‚0.12719827231485326 1loss/SRqqt1/detection_loss‚0.9611090969463641 2loss/FBt1/assignment_loss‚0.004605334372187069 /loss/FRt1/detection_loss‚1.5323038434698468 /loss/FBt2/detection_loss‚2.3517847110827765 trainer/global_step‚6899 /loss/FRt2/detection_loss‚2.3498188039151633 0loss/FBt2/assignment_loss‚0.4601587503918078 0loss/FBt4/assignment_loss‚0.1470227425340259 .loss/FBt4/detection_loss‚2.281996035150119¶9›¥“´‚(:ˆ ‡´åÔ›µ gpu.0.gpu‚64.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Ù¼”tµ‚(:é ‡´åÔ–ë¾· disk.drbd2266.out‚0" disk.dm-0.out‚58584.0185546875 disk.drbd1716.in‚0$ memory_percent‚99.57892894744873 proc.cpu.threads‚17 network.sent‚ 1.6805702e+07 disk.sda3.out‚ 261.08203125 disk.rbd0.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚58584.0185546875 disk.sda2.in‚0 disk.dm-0.in‚ 466.73828125 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 261.08203125 disk.nvme0n1.in‚ 466.73828125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875' disk.nvme0c0n1.out‚58584.0185546875 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 261.08203125! proc.memory.rssMB‚ 2959.453125* proc.memory.percent‚18.063068389892578 cpu‚25.258444215517706 network.recv‚ 2.307508e+06 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 466.73828125 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 68.98828125) disk./.usagePercent‚18.98657758642159$ disk./.usageGB‚282.9137268066406<²#眶‚(:‘ –´åԌҶ gpu.0.gpu‚65.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚301.22' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.24400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IJœÛs·‚(:è –´åÔøÅñ¸ disk.drbd2265.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda3.in‚0.203125 disk.sda3.out‚ 261.64453125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.sda.out‚ 261.64453125 disk.sda2.out‚0 network.sent‚ 1.6914209e+07! proc.memory.rssMB‚ 2959.453125 network.recv‚ 2.31378e+06$ disk./.usageGB‚282.9212875366211" disk.dm-0.out‚58654.5224609375 disk.drbd1716.out‚0% disk.nvme0n1.out‚58654.5224609375* proc.memory.percent‚18.063068389892578 cpu‚25.25802093071618 proc.cpu.threads‚17 disk.dm-1.out‚ 261.64453125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.73828125$ memory_percent‚99.57740306854248 disk.dm-0.in‚ 466.73828125 disk.drbd2080.in‚0 disk.sda2.in‚0* disk./.usagePercent‚18.987084993354454 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.73828125' disk.nvme0c0n1.out‚58654.5224609375' proc.memory.availableMB‚ 69.23828125=“¸‚(:ˆ ¥´åÔôûŠµ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚287.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýg³"q¹‚(:æ ¥´åÔÖ™¥· disk.drbd1716.out‚0% disk./.usageGB‚282.96077728271484 disk.nvme0n1.in‚ 466.73828125 disk.rbd1.out‚ 3.6796875 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚58746.64111328125 disk.dm-1.out‚ 262.296875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 262.296875 disk.drbd2265.in‚0 network.sent‚ 1.6917821e+07( disk.nvme0c0n1.out‚58746.64111328125 disk.sda3.in‚0.203125& proc.memory.availableMB‚ 68.9921875! proc.memory.rssMB‚ 2959.453125 disk.sda2.out‚0# disk.dm-0.out‚58746.64111328125 disk.drbd2080.out‚0 disk.sda.in‚0.203125 network.recv‚ 2.316222e+06" disk.nvme0c0n1.in‚ 466.73828125 disk.dm-0.in‚ 466.73828125 disk.sda1.out‚0* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 disk.sda.out‚ 262.296875) disk./.usagePercent‚18.98973518334891 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.dm-1.in‚0.203125 disk.rbd0.in‚0$ memory_percent‚99.57890510559082 cpu‚25.257586233661552 disk.drbd2265.out‚0 disk.rbd0.out‚0I1ì&2º- _wandbruntime‚6966  _runtime‚6966"OÊÇ»Á & lr-AdamW/pg1‚9.345137652256872e-05 & lr-AdamW/pg2‚9.345137652256872e-05 trainer/global_step‚6949 % _timestamp‚1.7884349883242548e+09 _runtime‚6966.679283146 _step‚567½ÀÄÌ¼Æ _runtime‚6966.679283146 _step‚567 & lr-AdamW/pg1‚9.345137652256872e-05 & lr-AdamW/pg2‚9.345137652256872e-05 trainer/global_step‚6949 % _timestamp‚1.7884349883242548e+09·²ð†<2½- _wandbruntime‚6966  _runtime‚6966GZ´Z(¾ˆP TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149466518 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚942147073049600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14840760335872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54467434 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚942147046658048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20592776814080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚956987833385472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚956984701600256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54466885 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚942147046658048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149489624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 $ _timestamp‚1.788434988325042e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚942147073049600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚940921360642878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124290287 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124290281 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14837654942208 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128773949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74306515 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚825618981096448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124290287 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚805026232416768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚940924489539770 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚956984701600256 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20592831828480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149466518 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚926108808061952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚926108782633984 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14815681477818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124290281 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128773397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74306512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14840760335872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273779911 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ŸÍÁ6Ò ¿P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124290287 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124290281 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14837654942208 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128773949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74306515 BDeviceStatsMonitor.on_train_batch’”l2A_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚825618981096448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124290287 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚805026232416768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚940924489539770 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚956984701600256 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20592831828480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149466518 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚926108808061952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚926108782633984 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14815681477818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124290281 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128773397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74306512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14840760335872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273779911 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149466518 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚942147073049600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14840760335872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54467434 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚942147046658048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20592776814080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚956987833385472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚956984701600256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54466885 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚942147046658048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149489624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 $ _timestamp‚1.788434988325042e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚942147073049600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚940921360642878¸w-2À- _wandbruntime‚6967  _runtime‚6967LÓ'Á‘N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74316765 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149510740 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149487608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54474344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚805138615146496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20595712105984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚926238205063168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124307565 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14817790271126 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚6949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚957118308002304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149510740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273818311 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124307571 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942278540746240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74316768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚825734410387968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273818311 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124307565 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14839767256064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20595767107072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚941055995334294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942278567137792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128791634 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54474866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14814686801994 # _timestamp‚1.78843498922479e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273795173 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚941052866437194 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚926238179635200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚ 6967.58072837 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚805138643280896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942278540746240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚569 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14842872663040 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128791109 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3‰%i'–N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54474866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14814686801994 # _timestamp‚1.78843498922479e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273795173 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚941052866437194 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚926238179635200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚ 6967.58072837 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚805138643280896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942278540746240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚569 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14842872663040 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128791109 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74316765 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149510740 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149487608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54474344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚805138615146496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20595712105984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚926238205063168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124307565 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14817790271126 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚6949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚957118308002304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149510740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273818311 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124307571 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942278540746240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74316768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚825734410387968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273818311 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124307565 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14839767256064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20595767107072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚941055995334294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942278567137792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128791634 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132¹eaîg2Ã- _wandbruntime‚6967  _runtime‚6967·sóô^ÄØ 0loss/FRt1/assignment_loss‚1.6237470640076532 3loss/FBt1/assignment_loss‚0.0005335826836257714 3loss/FBt3/assignment_loss‚0.0042008263988009276 /loss/FRt4/assignment_loss‚3.562953964356453 2loss/SRqqt2/assignment_loss‚1.4332882721315732 % _timestamp‚1.7884349892270067e+09 .loss/FRt4/detection_loss‚2.765009024450856 0loss/SRqqt4/detection_loss‚2.525964272433314 /loss/FBt4/detection_loss‚1.8655289581843786 /loss/FRt2/assignment_loss‚3.633841098965825 /loss/FRt3/assignment_loss‚3.137887737689874 /loss/FBt1/detection_loss‚0.4894911084448298 2loss/SRqqt4/assignment_loss‚1.9633986302490891 1loss/SRqqt2/detection_loss‚2.3377105557557307 epoch‚5 /loss/FRt3/detection_loss‚2.5089250665444593 1loss/SRqqt1/detection_loss‚0.8316872113494462 &loss/total_loss‚45.725142429814966 _step‚570 .loss/FRt2/detection_loss‚2.564235261968664 1loss/SRqqt3/assignment_loss‚1.320141812451184 1loss/FBt2/assignment_loss‚0.02451102192907447 1loss/FBt4/assignment_loss‚0.09415734317917668 /loss/FRt1/detection_loss‚1.7694411643204235 -loss/FBt2/detection_loss‚2.01928560091899 .loss/FBt3/detection_loss‚6.153583496809006 2loss/SRqqt1/assignment_loss‚1.1501585464644175 1loss/SRqqt3/detection_loss‚1.9454608081281186 trainer/global_step‚6949 _runtime‚6967.581960355–õÑcÅÝ 2loss/SRqqt4/assignment_loss‚1.9633986302490891 1loss/SRqqt2/detection_loss‚2.3377105557557307 epoch‚5 /loss/FRt3/detection_loss‚2.5089250665444593 1loss/SRqqt1/detection_loss‚0.8316872113494462 &loss/total_loss‚45.725142429814966 _step‚570 .loss/FRt2/detection_loss‚2.564235261968664 1loss/SRqqt3/assignment_loss‚1.320141812451184 1loss/FBt2/assignment_loss‚0.02451102192907447 1loss/FBt4/assignment_loss‚0.09415734317917668 /loss/FRt1/detection_loss‚1.7694411643204235 -loss/FBt2/detection_loss‚2.01928560091899 .loss/FBt3/detection_loss‚6.153583496809006 2loss/SRqqt1/assignment_loss‚1.1501585464644175 1loss/SRqqt3/detection_loss‚1.9454608081281186 trainer/global_step‚6949 _runtime‚6967.581960355 0loss/FRt1/assignment_loss‚1.6237470640076532 3loss/FBt1/assignment_loss‚0.0005335826836257714 3loss/FBt3/assignment_loss‚0.0042008263988009276 /loss/FRt4/assignment_loss‚3.562953964356453 2loss/SRqqt2/assignment_loss‚1.4332882721315732 % _timestamp‚1.7884349892270067e+09 .loss/FRt4/detection_loss‚2.765009024450856 0loss/SRqqt4/detection_loss‚2.525964272433314 /loss/FBt4/detection_loss‚1.8655289581843786 /loss/FRt2/assignment_loss‚3.633841098965825 /loss/FRt3/assignment_loss‚3.137887737689874 /loss/FBt1/detection_loss‚0.4894911084448298º|¢†/žÆ‚(:“ ´´åÔ¸ðû¶ gpu.0.gpu‚84.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.684999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rkNgÇ‚(:Ü ´´åÔÛ®ý¹ disk.sda.in‚0.203125 network.recv‚ 2.317601e+06 disk.dm-1.out‚ 262.7578125 disk.rbd1.in‚0 disk.sda3.out‚ 262.7578125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 466.73828125$ memory_percent‚99.57759380340576 proc.cpu.threads‚17 disk.dm-0.in‚ 466.73828125 network.sent‚ 1.692143e+07 disk.drbd2266.out‚0 disk.dm-0.out‚ 58811.8046875 disk.sda3.in‚0.203125 disk.sda1.in‚0! proc.memory.rssMB‚ 2959.453125 cpu‚25.25715711997723$ disk./.usageGB‚282.9586944580078 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 3.6796875$ disk.nvme0c0n1.out‚ 58811.8046875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0" disk.nvme0n1.out‚ 58811.8046875" disk.nvme0c0n1.in‚ 466.73828125 disk.drbd2080.out‚0 disk.sda.out‚ 262.7578125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 69.20703125* proc.memory.percent‚18.063068389892578* disk./.usagePercent‚18.989595403235214 disk.sda1.out‚0 disk.sda2.in‚04¶êžÈ‚(:“ ôåԼ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚211.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.226200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•0þÙwÉ‚(:ç ôåÔÙö¶) disk./.usagePercent‚18.99091998621737 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 263.390625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.6796875 disk.drbd2265.out‚0$ memory_percent‚99.57911968231201# disk.dm-0.out‚58883.31298828125 disk.sda2.in‚0 cpu‚25.256668750566828% disk./.usageGB‚282.97843170166016 disk.drbd1716.in‚0 disk.drbd2080.out‚0 network.recv‚ 2.322515e+06 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.nvme0n1.in‚ 466.73828125" disk.nvme0c0n1.in‚ 466.73828125 disk.sda3.out‚ 263.390625 disk.sda1.in‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚58883.31298828125 disk.dm-1.out‚ 263.390625 disk.drbd2266.in‚0' proc.memory.availableMB‚ 68.95703125 network.sent‚ 1.7028297e+07( disk.nvme0c0n1.out‚Ã×*û58883.31298828125 disk.drbd1716.out‚0* proc.memory.percent‚18.063068389892578 proc.cpu.threads‚17 disk.dm-0.in‚ 466.73828125 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125! proc.memory.rssMB‚ 2959.453125«O»¶“Ê‚(:ˆ Ò´åÔÞÏ¿µ gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚284.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pxÛÿwË‚(:ì Ò´åÔÞ¬ì¶ disk.dm-0.in‚ 466.90234375 cpu‚25.25626094275347) disk./.usagePercent‚18.99201569846392 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 263.73828125 disk.rbd0.in‚0 disk.sda2.in‚0# disk.dm-0.out‚58954.51611328125' proc.memory.availableMB‚ 68.39453125 network.sent‚ 1.7031575e+07 disk.sda3.out‚ 263.73828125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125$ memory_percent‚99.58255290985107* proc.memory.percent‚18.063068389892578 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 263.73828125 disk.drbd2266.out‚0 network.recv‚ 2.324379e+06 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚58954.51611328125! proc.memory.rssMB‚ 2959.453125 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 466.90234375 proc.cpu.threads‚17% disk./.usageGB‚282.99475860595703& disk.nvme0n1.out‚58954.51611328125‘5N2Ì- _wandbruntime‚7014  _runtime‚7014Ø>¤€ÇÍÁ & lr-AdamW/pg1‚9.198126170525328e-05 & lr-AdamW/pg2‚9.198126170525328e-05 trainer/global_step‚6999 % _timestamp‚1.7884350361322958e+09 _runtime‚7014.487334974 _step‚571¶ &ÌÎÆ _step‚571 & lr-AdamW/pg1‚9.198126170525328e-05 & lr-AdamW/pg2‚9.198126170525328e-05 trainer/global_step‚6999 % _timestamp‚1.7884350361322958e+09 _runtime‚7014.487334974»åB‰2Ï- _wandbruntime‚7014  _runtime‚7014bü!(ЉP eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚810646775623168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788435036133099e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54849679 trainer/global_step‚6999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948721777459200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275701963 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125154481 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129668299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125154487 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚932578632694784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚963665036116480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74819165 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74819162 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14918031255102 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129668844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150524370 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275678851 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150547476 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150524370 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚963668167901696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948721777459200 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚947499792846778 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125154481 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150547476 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14921134724026 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚831385986109440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚948721751067648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125154487 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚947496663949886 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚810646747488768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚7014.488428578 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚963668167901696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20739293635072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948721751067648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54849137 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚831386069258240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963665036116480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14943285048832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14943285048832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚932578658122752 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14946390442496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚947499792846778 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125154481 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150547476 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14921134724026 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚831385986109440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚948721751067648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125154487 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚947496663949886 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚810646747488768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚7014.488428578 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚963668167901696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20739293635072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948721751067648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54849137 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚831386069258240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963665036116480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14943285048832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14943285048832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚932578658122752 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14946390442496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129668299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125154487 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚932578632694784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚963665036116480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664¼Êað2Ò- _wandbruntime‚7015  _runtime‚7015k\ÉŒb&Ó”N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125171771 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129686621 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129686090 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275717225 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74829418 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275740363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275717225 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948853271547392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150568592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74829415 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚932708029696000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150545460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963801774317056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125171771 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948853245155840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚831501334709248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚947631298641302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚932708055123968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚6999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14945397362688 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150568592 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚810759186487296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20742231357440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125171765 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275740363 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14948502769664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚810759158352896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948853271547392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚831501417844736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.smaláê{¸l_pool.freed‚54856675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚947628169744202 _step‚573«~×M'Ô™N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125171771 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948853245155840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚831501334709248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚947631298641302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚932708055123968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚6999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14945397362688 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150568592 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚810759186487296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20742231357440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125171765 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275740363 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14948502769664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚810759158352896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948853271547392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚831501417844736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54856675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚947628169744202 _step‚573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125171771 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129686621 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129686090 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275717225 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74829418 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275740363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275717225 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948853271547392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150568592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74829415 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚932708029696000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150545460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963801774317056½1 pê2Õ- _wandbruntime‚7015  _runtime‚7015ìÆ+i]Ö× % _timestamp‚1.7884350370521905e+09 /loss/FRt3/assignment_loss‚3.097938890116555 .loss/FRt4/detection_loss‚2.725170058423075 /loss/FRt2/assignment_loss‚3.173287179734972 1loss/SRqqt2/detection_loss‚2.0932025573192496 _runtime‚7015.407173061 2loss/SRqqt4/assignment_loss‚1.3603576502722243 0loss/FBt3/assignment_loss‚0.1879555977647449 .loss/FRt1/detection_loss‚1.910165390620629 /loss/FBt3/detection_loss‚1.8959954593862807 epoch‚5 /loss/FRt1/assignment_loss‚1.463910946139583 2loss/SRqqt3/assignment_loss‚1.0629205346489563 .loss/FBt2/detection_loss‚2.247032587017332 2loss/SRqqt1/assignment_loss‚0.8966818974731723 2loss/SRqqt2/assignment_loss‚1.4072904242919042 /loss/FRt3/detection_loss‚2.7420590136732375 1loss/SRqqt3/detection_loss‚2.2598810348755274 0loss/SRqqt4/detection_loss‚2.818780629531197 /loss/FBt4/detection_loss‚1.9553399431078058 _step‚574 /loss/FRt4/assignment_loss‚3.544299910808432 1loss/FBt2/assignment_loss‚0.09425851512891573 1loss/FBt4/assignment_loss‚0.18516730159657718 /loss/FRt2/detection_loss‚2.2643391331036886 &loss/total_loss‚40.821632548309566 trainer/global_step‚6999 2loss/FBt1/assignment_loss‚0.006480994677355745 0loss/FBt1/detection_loss‚0.49742418457753956 1loss/SRqqt1/detection_loss‚0.9316927140206098 ˜ b×Ü 2loss/FBt1/assignment_loss‚0.006480994677355745 0loss/FBt1/detection_loss‚0.49742418457753956 1loss/SRqqt1/detection_loss‚0.9316927140206098 % _timestamp‚1.7884350370521905e+09 /loss/FRt3/assignment_loss‚3.097938890116555 .loss/FRt4/detection_loss‚2.725170058423075 /loss/FRt2/assignment_loss‚3.173287179734972 1loss/SRqqt2/detection_loss‚2.0932025573192496 _runtime‚7015.407173061 2loss/SRqqt4/assignment_loss‚1.3603576502722243 0loss/FBt3/assignment_loss‚0.1879555977647449 .loss/FRt1/detection_loss‚1.910165390620629 /loss/FBt3/detection_loss‚1.8959954593862807 epoch‚5 /loss/FRt1/assignment_loss‚1.463910946139583 2loss/SRqqt3/assignment_loss‚1.0629205346489563 .loss/FBt2/detection_loss‚2.247032587017332 2loss/SRqqt1/assignment_loss‚0.8966818974731723 2loss/SRqqt2/assignment_loss‚1.4072904242919042 /loss/FRt3/detection_loss‚2.7420590136732375 1loss/SRqqt3/detection_loss‚2.2598810348755274 0loss/SRqqt4/detection_loss‚2.818780629531197 /loss/FBt4/detection_loss‚1.9553399431078058 _step‚574 /loss/FRt4/assignment_loss‚3.544299910808432 1loss/FBt2/assignment_loss‚0.09425851512891573 1loss/FBt4/assignment_loss‚0.18516730159657718 /loss/FRt2/detection_loss‚2.2643391331036886 &loss/total_loss‚40.821632548309566 trainer/global_step‚6999¾<Ìh¦žØ‚(:“ á´åÔÏÀ†µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚290.708' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.141600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÌð×tÙ‚(:é á´åÔÂæó¶ disk.dm-0.in‚ 466.90234375 disk.drbd2080.out‚0 disk.sda.in‚0.203125* proc.memory.percent‚18.063068389892578 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0" disk.dm-0.out‚59043.4287109375% disk./.usageGB‚283.01361083984375 disk.sda2.in‚0 disk.nvme0n1.in‚ 466.90234375% disk.nvme0n1.out‚59043.4287109375' disk.nvme0c0n1.out‚59043.4287109375 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.dm-1.out‚ 264.48046875 network.sent‚ 1.703784e+07 network.recv‚ 2.328581e+06* disk./.usagePercent‚18.993280887698138 disk.rbd0.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda3.out‚ 264.48046875 disk.drbd2265.out‚0" disk.nvme0c0n1.in‚ 466.90234375$ memory_percent‚99.58274364471436! proc.memory.rssMB‚ 2959.453125 cpu‚25.25582140436605 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda.out‚ 264.48046875 proc.cpu.threads‚17 disk.drbd2080.in‚0' proc.memory.availableMB‚ 68.36328125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0xƒ0Ú‚(:’ ð´åÔÁÞµ gpu.0.gpu‚45.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.78' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.955999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N#ŠÑmÛ‚(:â ð´åÔĤ¶ disk.drbd2265.out‚0% disk.nvme0n1.out‚59109.2373046875 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ disk./.usageGB‚283.0264015197754 disk.sda3.out‚ 264.828125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚59109.2373046875 disk.drbd2080.out‚0 cpu‚25.255387054676646 network.recv‚ 2.332575e+06 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 network.sent‚ 1.7143435e+07! proc.memory.rssMB‚ 2959.453125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 264.828125" disk.dm-0.out‚59109.2373046875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 264.828125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.6796875 disk.rbd0.in‚0 disk.dm-0.in‚ 466.90234375$ memory_percent‚99.58276748657227% proc.memory.availableMB‚ 68.359375* proc.memory.percent‚18.063068389892578* disk./.usagePercent‚18.994139280960447 disk.drbd2080.in‚0 disk.sda.in‚0.203125ñv ΓÜ‚(:ˆ ÿ´åÔæÁµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸fmÝ‚(:â ÿ´åÔ؈ì¶$ disk.nvme0n1.out‚59182.294921875 disk.dm-1.out‚ 265.5390625 disk.rbd1.out‚ 3.6796875$ memory_percent‚99.58276748657227 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375 network.sent‚ 1.7148694e+07! disk.dm-0.out‚59182.294921875 disk.sda3.out‚ 265.5390625 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚59182.294921875 network.recv‚ 2.335636e+06 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2959.453125 disk.drbd2080.out‚0 cpu‚25.25494919678329 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0% proc.memory.availableMB‚ 68.359375* proc.memory.percent‚18.063068389892578 disk.sda1.out‚0* disk./.usagePercent‚18.994978985672752% disk./.usageGB‚283.03891372680664 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.sda.out‚ 265.5390625 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 466.90234375ô™ˆX2Þ- _wandbruntime‚7062  _runtime‚7062ÏÒ8+ÆßÀ trainer/global_step‚7049 % _timestamp‚1.7884350845200903e+09 _runtime‚ 7062.87511778 _step‚575 & lr-AdamW/pg1‚9.051329421600652e-05 & lr-AdamW/pg2‚9.051329421600652e-05Að¢zËàÅ % _timestamp‚1.7884350845200903e+09 _runtime‚ 7062.87511778 _step‚575 & lr-AdamW/pg1‚9.051329421600652e-05 & lr-AdamW/pg2‚9.051329421600652e-05 trainer/global_step‚7049¿×W´‘2á- _wandbruntime‚7062  _runtime‚7062À\í´(âŠP N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277624015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55229348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126018687 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚816267290695168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126018687 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚954075096153786 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚7062.876032296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970348502417920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277600903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚955296481868800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130561700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚837152980140544 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151605328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970345370632704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15052020549120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15048915155456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126018681 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126018681 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151605328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚939048482755584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚955296455477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75331815 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20885606296576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15052020549120 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130561160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚970345370632704 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚837152896991744 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970345370632704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15052020549120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15048915155456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126018681 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126018681 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151605328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚939048482755584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚955296455477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75331815 `MDeviceStatsMonitor.on_train_batc˜œòÕuh_start/inactive_split_bytes.small_pool.freed‚20885606296576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15052020549120 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130561160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚970345370632704 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚837152896991744 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚7062.876032296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970348502417920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277600903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚955296481868800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130561700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚837152980140544 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151605328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151603312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚837268308546048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277639277 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151626444 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20888578852352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15051027469312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 % _timestamp‚1.7884350854404423e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55236854 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130579448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75342068 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚816379701559296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚577 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15054132876288 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚954206601948310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚955427975956992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126035965 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚970482108833280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚955427949565440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130578919 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75342065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚939177879756800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15028696763542 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚939177905184768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277662415 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚955427975956992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚7063.796512302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970482108833280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151626444 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚837268223313408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151603312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚837268308546048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277639277 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151626444 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20888578852352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15051027469312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 % _timestamp‚1.7884350854404423e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55236854 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130579448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75342068 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚816379701559296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚577 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15054132876288 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚954206601948310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚955427975956992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126035965 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚970482108833280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚955427949565440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130578919 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75342065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚939177879756800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15028696763542 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚939177905184768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277662415 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚955427975956992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚7063.796512302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970482108833280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151626444 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚837268223313408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279546067 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126882881 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126882887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚945518358244352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15132041216442 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚977028836934144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _step‚580 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152640074 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15154545262080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚961871186278400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚961871186278400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚842919819290624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55610267 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚960650399460794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚977028836934144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚821887833901568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152640074 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152663180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126882887 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152663180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 trainer/global_step‚7099 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55609727 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131454732 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚977025705148928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945518332816384 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15154545262080 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279546067 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126882881 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126882887 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚945518358244352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15132041216442 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚977028836934144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _step‚580 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152640074 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15154545262080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚961871186278400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚961871186278400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚842919819290624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55610267 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚960650399460794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚977028836934144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚821887833901568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152640074 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152663180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126882887 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152663180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 trainer/global_step‚7099 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55609727 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131454732 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚977025705148928Ä¿ËX!2ø- _wandbruntime‚7111  _runtime‚7111•kùl&$ù”N RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152661164 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75854718 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126900165 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279584467 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126900165 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152684296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 trainer/global_step‚7099 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15131046540618 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75854715 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚977159311550976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚977162443349504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚945647729817600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131471921 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152684296 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279561329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55617206 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚822000272900096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚822000244765696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚960781905255318 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15159762982912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚843035245578752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126900171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962002680366592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7111.833080592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15156657575936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152661164 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15159762982912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131472451 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279584467 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚945647755245568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126900171 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843035160346112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884351334773166e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚581 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15156657575936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚960778776358218 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15134150009750 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15159762982912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131472451 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279584467 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚945647755245568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126900171 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843035160346112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884351334773166e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚581 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15156657575936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚960778776358218 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15134150009750 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152661164 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75854718 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126900165 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279584467 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126900165 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152684296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 trainer/global_step‚7099 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15131046540618 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75854715 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚977159311550976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚977162443349504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚945647729817600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131471921 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152684296 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279561329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55617206 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚822000272900096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚822000244765696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚960781905255318 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15159762982912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚843035245578752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126900171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962002680366592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7111.833080592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15156657575936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152661164 B*T(ˆ‰P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚967222573870910 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚983709171450368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55990508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚983706039665152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153697926 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21178177362432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15234390993726 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15263280762368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281445007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚968445864296448 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281468119 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15237494462650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127747087 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127747081 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132347062 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884351806673913e+09 _step‚584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55989950 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚951988208305152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76357112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153721032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15260175368704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚968445890688000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127747087 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚827508405242368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚968445890688000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚968445864296448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21178234473984Y(€"k‰ŽP _step‚584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55989950 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚951988208305152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76357112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153721032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15260175368704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚968445890688000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127747087 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚827508405242368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚968445890688000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚968445864296448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21178234473984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚967222573870910 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚983709171450368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55990508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚983706039665152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153697926 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21178177362432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15234390993726 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15263280762368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281445007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚968445864296448 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281468119 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15237494462650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127747087 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127747081 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132347062 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884351806673913e+09ÈÊù`A2Š- _wandbruntime‚7159  _runtime‚7159“®'‹”N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55997598 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21181099392000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚952117579878400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21181154393088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281483381 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153742148 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15265393089536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884351815726497e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚968577358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281506519 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚983842777865728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127764371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127764365 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚827620816106496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281506519 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132365497 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚968577358384640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76367368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚585 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚968577384776192 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132364963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55998129 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15239603255958 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153719016 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127764371 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127764365 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15262287682560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚848801887364096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232¨žö'Œ™N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848801970499584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15236499786826 _runtime‚7159.928254964 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153719016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76367365 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132364963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55998129 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15239603255958 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153719016 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127764371 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127764365 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15262287682560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚848801887364096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55997598 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21181099392000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚952117579878400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21181154393088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281483381 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153742148 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15265393089536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884351815726497e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚968577358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281506519 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚983842777865728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127764371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127764365 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚827620816106496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281506519 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132365497 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚968577358384640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76367368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚585 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚968577384776192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚990386374181376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚990389505966592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚7199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚958458058365952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76869765 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚833128948448768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚973801006074810 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚854453841046016 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128611281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚990386374181376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128611287 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚975020568706048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975020568706048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128611287 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚958458058365952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76869765 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚833128948448768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚973801006074810 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚854453841046016 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128611281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚990386374181376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128611287 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚975020568706048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975020568706048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128611287 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚990386374181376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚990389505966592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚7199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15345056502166 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283428571 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128628565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76880015 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚975152089185792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56381395 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚854569115081728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283405433 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚958587429939200 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128628571 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283428571 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚958587455367168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128628565 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15371023196160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15341953033034 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚990523112381952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154800000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884352295995097e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15371023196160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133261410 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚833241359312896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456Œ÷U Øž™N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7207.954988594 ?5DeviceStatsMonitor.on_train_batch_€Œ›G&end/active.all.peak‚28217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚990523112381952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚990519980583424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚975152062794240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚854569196120064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚975152089185792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128628571 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21327783903232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154776868 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21327836807168 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283405433 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154800000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154800000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884352295995097e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15371023196160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133261410 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚833241359312896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15345056502166 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283428571 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128628565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76880015 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚975152089185792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56381395 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚854569115081728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283405433 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚958587429939200 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128628571 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283428571 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚958587455367168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128628565 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15371023196160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15341953033034 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚990523112381952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚997066708697600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15474540975616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77382415 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚964927882998784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚860220849718784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129475487 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134137251 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚838749463520768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155813630 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981595299507200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚997069840482816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155836736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚964927908426752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129475487 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15471435581952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚7255.043608475 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155813630 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134136700 % _timestamp‚1.7884352766881902e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15471435581952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚981595299507200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15474540975616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚997066708697600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚981595273115648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 285312223 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981595273115648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129475481 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21471443309568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21471386198016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77382412 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚860220934964736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚997069840482816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 trainer/global_step‚7249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155836736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚997066708697600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚981595273115648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 285312223 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981595273115648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129475481 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21471443309568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21471386198016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77382412 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚860220934964736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚997069840482816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 trainer/global_step‚7249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155836736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚997066708697600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15474540975616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77382415 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚964927882998784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚860220849718784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129475487 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134137251 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚838749463520768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155813630 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981595299507200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚997069840482816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155836736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚964927908426752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129475487 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15471435581952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚7255.043608475 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155813630 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134136700 % _timestamp‚1.7884352766881902e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15471435581952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚981595299507200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ð'“#2®- _wandbruntime‚7255  _runtime‚7255þçL'¯”N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15473547895808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚860336188845056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155834720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15476653302784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚860336274077696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884352776216486e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚965057305427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚965057280000000 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚997203446898176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285350623 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285350623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚981726767203840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77392665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚7255.977204488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285327485 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21474314460160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77392668 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚981726767203840 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134154571 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚980504686279242 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚7249¾,@o'°™N BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚981726767203840 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134154571 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚980504686279242 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚7249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15473547895808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚860336188845056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155834720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15476653302784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚860336274077696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884352776216486e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚965057305427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚965057280000000 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚997203446898176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285350623 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285350623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚981726767203840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77392665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚7255.977204488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285327485 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21474314460160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77392668 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚144 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143806976 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 156318124 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 130208609 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15509518558208 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 156299846 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚969737423845376 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18278 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1002149972485120 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚985220709547938 _runtime‚7315.307626536 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚986642807827968 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 130208604 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚146 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚985218358526282 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 286526733 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚864511156429824 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15483285702562 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 130208609 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 134926505 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 130208604 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 156299846 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1002152326386176 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7253 _step‚595 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1700 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚969737400514560 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚15507188951552 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 286508450 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚971 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚144 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143806976 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0Óq³[2¿- _wandbruntime‚7316  _runtime‚7316i§umÀ"ÀS KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚1002242417013248 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 130221802 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _step‚596 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚77916878 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 28510781440 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15481454463370 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1002242417013248 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚986732376143360 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 134940814 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1002240063112192 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚146 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚969822373445632 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15510016575488 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15510016575488 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 286529342 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚969822350114816 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚15483782154210 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 % _timestamp‚1.7884353380441608e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 286529342 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚986732376143360 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 126158336 _runtime‚7316.399667722 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 156325818 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1002240063112192 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 hTDeviceStatsMonitor.on_validation_batch_end/iª‡ÎÇnactive_split_bytes.large_pool.allocated‚843075382238208 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 130221807 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚57023934 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚864587074979328 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚15507686968832 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18278 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878ÂOŒ)Á†S PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1302 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2353901056 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚843075364589568 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021656 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 trainer/global_step‚7253 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚986732400437760 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 130221807 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143806976 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 130221802 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 156325818 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚144 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 156307540 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1302 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚15507686968832 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 286547625 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚57023790 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3294746624 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚21511566582784 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚985306155599842 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606656 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18283 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15510016575488 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15510016575488 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 286529342 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚969822350114816 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚15483782154210 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 % _timestamp‚1.7884353380441608e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 286529342 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚986732376143360 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 126158336 _runtime‚7316.399667722 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 156325818 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1002240063112192 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚843075382238208 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 130221807 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚57023934 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚864587074979328 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚15507686968832 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18278 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚1002242417013248 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 130221802 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _step‚596 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚77916878 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 28510781440 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15481454463370 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1002242417013248 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚986732376143360 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 134940814 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1002240063112192 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚146 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚969822373445632 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42763158679008484 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025078391656279564 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2222222089767456 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375000298023224 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_3‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.025856638327240944 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.27969351410865784 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3377857804298401 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38463714718818665 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9508383870124817 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.034754227846860886 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010129956528544426 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2738725244998932 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15469685196876526 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5442012548446655 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2738749384880066 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887279272079468 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36019930243492126 *particle/sensitivity‚0.286043643951416 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4912281036376953 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.30157962441444397 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36289453506469727 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5065173506736755 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3405172526836395 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.46309521794319153 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29983463883399963 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3439665138721466 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18196934461593628 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30332517623901367 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04022988677024841 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01320754736661911 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *"Purity/*FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18196934461593628 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_1_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9893602132797241 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.38394665718078613 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1509518176317215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48036548495292664 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20996712148189545 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.28463685512542725 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23639455437660217 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5464743971824646 ,jet/accuracy_3_of_3‚0.024738145992159843 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2600586414337158 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2850532829761505 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.46790292859077454 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3218167722225189 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2787298262119293 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1353546679019928 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2222222238779068 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24872715771198273 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08618708699941635 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42158639430999756 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.005820105783641338 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26178160309791565 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4820230305194855 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0020674937404692173 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32314813137054443 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.285380482673645 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.287555456161499 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32269376516342163 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.30148473381996155 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9895942807197571 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3472222089767456 *jet/accuracy_2_of_6‚0.5128205418586731 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12493742257356644 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.00898266863077879 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37708336114883423 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.34090912342071533 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2780698835849762 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3498679995536804 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982104480266571 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375000298023224 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14016179740428925 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3368420898914337 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2971075475215912 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3009170889854431 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3189188838005066 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3387470841407776 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2856321632862091 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6305297613143921 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23701706528663635 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.009173768572509289 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2614240348339081 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15592657029628754 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3394790291786194 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03551478311419487 particle/accuracy_2_of_5‚1 jet/accuracy_5_of_7‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36627906560897827 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 particle/accuracy_7_of_7‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2657296359539032 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2896745204925537 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9846101999282837 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 trainer/global_step‚7253 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1879432499408722 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt3SRqqt*FBtvxÓBÝI/SRqqt_purity‚0.29490700364112854 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 _runtime‚7331.217640278 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5511585474014282 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28474727272987366 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2059372365474701 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 +"Purity/1FRt0SRqqt3FBt/event_purity‚0.25 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3540703356266022 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.27272728085517883 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3745884597301483 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.32870370149612427 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3522837460041046 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.319819837808609 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882344007492065 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20515979826450348 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4746910035610199 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.24666665494441986 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 particle/accuracy_4_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31976884603500366 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.34402233362197876 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873809218406677 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3198276162147522 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.28387799859046936 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 (particle/f_score‚0.37258172035217285 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903764128684998 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48717159032821655 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3920981287956238 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.098565474152565 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38352158665657043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2979685664176941 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903210997581482 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880803823471069 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.023769794031977654 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2510901987552643 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.28907090425491333 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2861989438533783 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3464917838573456 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930981993675232 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 epoch‚5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9917840361595154 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3809644281864166 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3923884928226471 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9925946593284607 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2391345500946045 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *loss/val_total_loss‚43.369728625415874 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_6_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34453538060188293 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24623796343803406 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4436657726764679 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2558938264846802 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.295879602432251 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3423670530319214 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.694165050983429 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.26666662096977234 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18148505687713623 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3640211522579193 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4166666865348816 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5017115473747253 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43885108828544617 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2971075475215912 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/2FRt1SRqqt0FBt/event_purity‚0.00257936492562294 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2885521650314331 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9957671761512756 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5281355977058411 *jet/accuracy_1_of_4‚0.8778786659240723 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4236111342906952 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35232996940612793 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26403164863586426 particle/accuracy_2_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2730734944343567 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47246232628822327 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4027743637561798 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.29440197348594666 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.27680209279060364 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.283430814743042 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3022707402706146 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5736110806465149 ,jet/accuracy_4_of_5‚0.006496415939182043 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5000000596046448 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35010406374931335 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37812504172325134 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.304417222738266 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05922619253396988 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6904599666595459 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49523037672042847 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +particle/specificity‚0.9539421200752258 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983122944831848 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18490831553936005 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.6000000238418579 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333134651184 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_7‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 _step‚597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8247343301773071 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.2921811044216156 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2344820201396942 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975050091743469 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9933657646179199 *jet/accuracy_1_of_3‚0.8262971639633179 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.06779684126377106 particle/accuracy_3_of_3‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38188040256500244 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09904472529888153 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3150903880596161 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +jet/accuracy_3_of_5‚0.09110342711210251 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49836844205856323 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018216747790575027 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.42220941185951233 particle/accuracy_2_of_6‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24228395521640778 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3028036952018738 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010843115858733654 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 particle/accuracy_3_of_6‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4811892509460449 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008368952199816704 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.020993422716856003 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.44109195470809937 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3538690507411957 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014491314068436623 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3053776025772095 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4781745672225952 +jet/accuracy_2_of_2‚0.09656877815723419 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23777776956558228 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 % _timestamp‚1.7884353528605235e+09 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9687500596046448 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375nA”É6Æ©¦ & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014491314068436623 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3053776025772095 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4781745672225952 +jet/accuracy_2_of_2‚0.09656877815723419 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23777776956558228 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 % _timestamp‚1.7884353528605235e+09 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9687500596046448 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 +jet/accuracy_2_of_3‚0.22283092141151428 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_6_of_6‚1 jet/accuracy_7_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918574094772339 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07951587438583374 +jet/accuracy_1_of_1‚0.39838290214538574 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963807463645935 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.35478711128234863 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 'particle/accuracy‚0.849640429019928 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15425637364387512 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24655470252037048 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333134651184 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007289170753210783 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2515440285205841 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3208323121070862 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42763158679008484 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025078391656279564 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2222222089767456 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375000298023224 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_3‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.025856638327240944 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.27969351410865784 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3377857804298401 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38463714718818665 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9508383870124817 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.289814829826355 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.034754227846860886 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010129956528544426 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2738725244998932 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15469685196876526 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5442012548446655 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2738749384880066 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887279272079468 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36019930243492126 *particle/sensitivity‚0.286043643951416 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4912281036376953 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.30157962441444397 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36289453506469727 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5065173506736755 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3405172526836395 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.46309521794319153 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29983463883399963 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3439665138721466 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18196934461593628 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30332517623901367 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04022988677024841 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01320754736661911 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *"Purity/*FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18196934461593628 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_1_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9893602132797241 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.38394665718078613 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1509518176317215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48036548495292664 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20996712148189545 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.28463685512542725 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23639455437660217 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5464743971824646 ,jet/accuracy_3_of_3‚0.024738145992159843 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2600586414337158 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2850532829761505 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.46790292859077454 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3218167722225189 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2787298262119293 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1353546679019928 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2222222238779068 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24872715771198273 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08618708699941635 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42158639430999756 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.005820105783641338 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26178160309791565 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4820230305194855 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0020674937404692173 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32314813137054443 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.285380482673645 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.287555456161499 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32269376516342163 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.30148473381996155 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9895942807197571 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & PuÁG{]rity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3472222089767456 *jet/accuracy_2_of_6‚0.5128205418586731 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12493742257356644 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.00898266863077879 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37708336114883423 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.34090912342071533 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2780698835849762 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3498679995536804 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982104480266571 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375000298023224 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14016179740428925 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3368420898914337 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2971075475215912 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3009170889854431 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3189188838005066 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3387470841407776 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2856321632862091 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6305297613143921 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23701706528663635 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.009173768572509289 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2614240348339081 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15592657029628754 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3394790291786194 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03551478311419487 particle/accuracy_2_of_5‚1 jet/accuracy_5_of_7‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36627906560897827 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 particle/accuracy_7_of_7‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2657296359539032 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2896745204925537 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9846101999282837 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 trainer/global_step‚7253 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1879432499408722 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.29490700364112854 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 _runtime‚7331.217640278 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5511585474014282 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28474727272987366 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2059372365474701 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 +"Purity/1FRt0SRqqt3FBt/event_purity‚0.25 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3540703356266022 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.27272728085517883 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3745884597301483 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.32870370149612427 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3522837460041046 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.319819837808609 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882344007492065 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20515979826450348 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4746910035610199 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.24666665494441986 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 particle/accuracy_4_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31976884603500366 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.34402233362197876 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873809218406677 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3198276162147522 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.28387799859046936 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 (particle/f_score‚0.37258172035217285 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903764128684998 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48717159032821655 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3920981287956238 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.098565474152565 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38352158665657043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2979685664176941 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903210997581482 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880803823471069 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.023769794031977654 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2510901987552643 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.28907090425491333 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2861989438533783 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3464917838573456 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930981993675232 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 epoch‚5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9917840361595154 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3809644281864166 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3923884928226471 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9925946593284607 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2391345500946045 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *loss/val_total_loss‚43.369728625415874 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_6_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34453538060188293 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24623796343803406 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4436657726764679 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2558938264846802 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.295879602432251 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3423670530319214 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.694165050983429 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.26666662096977234 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18148505687713623 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3640211522579193 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4166666865348816 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5017115473747253 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43885108828544617 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2971075475215912 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/2FRt1SRqqt0FBt/event_purity‚0.00257936492562294 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2885521650314331 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9957671761512756 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5281355977058411 *jet/accuracy_1_of_4‚0.8778786659240723 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4236111342906952 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35232996940612793 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26403164863586426 particle/accuracy_2_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2730734944343567 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47246232628822327 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4027743637561798 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.29440197348594666 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.27680209279060364 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.283430814743042 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3022707402706146 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5736110806465149 ,jet/accuracy_4_of_5‚0.006496415939182043 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5000000596046448 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35010406374931335 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37812504172325134 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.304417222738266 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05922619253396988 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6904599666595459 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49523037672042847 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +particle/specificity‚0.9539421200752258 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983122944831848 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18490831553936005 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.6000000238418579 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333134651184 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_7‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 _step‚597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8247343301773071 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.2921811044216156 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2344820201396942 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975050091743469 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9933657646179199 *jet/accuracy_1_of_3‚0.8262971639633179 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.06779684126377106 particle/accuracy_3_of_3‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38188040256500244 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09904472529888153 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3150903880596161 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +jet/accuracy_3_of_5‚0.09110342711210251 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49836844205856323 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018216747790575027 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.42220941185951233 particle/accuracy_2_of_6‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24228395521640778 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3028036952018738 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010843115858733654 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 particle/accuracy_3_of_6‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4811892509460449 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008368952199816704 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.020993422716856003 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.44109195470809937 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3538690507411957 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1Õ%;o“Ç‚(:ˆ œ·åÔ³§µ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚305.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.0572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£éžuÈ‚(:ê œ·åÔ· disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚60627.40478515625 disk.sda3.in‚0.203125# memory_percent‚99.4899034500122 network.sent‚ 1.7869931e+07 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 275.8671875 disk.rbd0.out‚0) proc.memory.percent‚17.91222095489502* disk./.usagePercent‚19.018779238108472 disk.drbd2266.out‚0 network.recv‚ 2.428949e+06 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.8603515625" disk.nvme0c0n1.in‚ 466.90234375 cpu‚25.290078517102415 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚60627.40478515625' proc.memory.availableMB‚ 83.57421875# proc.memory.rssMB‚ 2934.73828125 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 275.8671875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚60627.40478515625 disk.sda3.out‚ 275.8671875 proc.cpu.threads‚17! disk./.usageGB‚283.3935546875D…7’É‚(:‡ «·åÔÛëµ gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚254.525' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.905 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üH½|Ê‚(:ñ «·åÔúÚζ( disk.nvme0c0n1.out‚60725.61181640625 disk.dm-0.in‚ 466.90234375* proc.memory.percent‚17.926621437072754 cpu‚25.289593484764623 proc.cpu.threads‚17 disk.nvme0n1.in‚ 466.90234375 disk.sda3.out‚ 276.48828125 disk.sda1.out‚0 disk.drbd2265.out‚0& proc.memory.availableMB‚ 84.5546875) disk./.usagePercent‚19.02080169762897 disk.drbd1716.out‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2937.09765625 disk.rbd0.in‚0 disk.dm-1.out‚ 276.48828125 disk.rbd1.out‚ 3.8603515625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.out‚ 276.48828125 disk.rbd1.in‚0# disk.dm-0.out‚60725.61181640625 disk.sda3.in‚0.203125 network.recv‚ 2.449531e+06 disk.sda.in‚0.203125 network.sent‚ 1.8000578e+07% disk./.usageGB‚283.42369079589844 disk.rbd0.out‚0& disk.nvme0n1.out‚60725.61181640625" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125$ memory_percent‚99.48391914367676 disk.drbd2266.in‚0 disk.drbd2266.out‚0E1øåžË‚(:“ º·åÔ÷üø´ gpu.0.gpu‚66.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚293.084' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.616800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ZŒ|Ì‚(:ñ º·åÔ‰¨Î·# disk.dm-0.out‚60810.67041015625 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 3.92333984375 disk.sda3.out‚ 277.05859375 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2940.14453125 cpu‚25.289133945590557 network.recv‚ 2.465212e+06) disk./.usagePercent‚19.02341604656863# disk./.usageGB‚283.462646484375 disk.rbd1.in‚0 disk.dm-0.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 277.05859375* proc.memory.percent‚17.945218086242676 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0$ memory_percent‚99.47168827056885 network.sent‚ 1.8012134e+07 disk.drbd2080.out‚0 disk.sda.out‚ 277.05859375" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚60810.67041015625 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚60810.67041015625 disk.drbd2266.out‚0' proc.memory.availableMB‚ 86.55859375G g2Í- _wandbruntime‚7374  _runtime‚7374¨ÖçRÇÎÁ _runtime‚7374.299776447 _step‚598 & lr-AdamW/pg1‚8.321635473904348e-05 & lr-AdamW/pg2‚8.321635473904348e-05 trainer/global_step‚7299 % _timestamp‚1.7884353959447103e+09õ–ùÕÌÏÆ _runtime‚7374.299776447 _step‚598 & lr-AdamW/pg1‚8.321635473904348e-05 & lr-AdamW/pg2‚8.321635473904348e-05 trainer/global_step‚7299 % _timestamp‚1.7884353959447103e+09ÖøoS2Ð- _wandbruntime‚7374  _runtime‚7374Hæ†jÑŽP RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135932177 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚992331602169259 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15611563159040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚993814307840000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78503887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚870763838920192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚976749583303680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15582044293547 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57427769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131171155 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_staª*©áªrt/reserved_bytes.small_pool.allocated‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1009425897390592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15611563159040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135931656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009422765605376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚7374.300746948 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78503890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157358136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993814334231552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15608457765376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131171155 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157381242 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚849112484110848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚992334731066151 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1009422765605376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚976749557875712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157381242 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009425897390592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21651382943744 _step‚599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚993814334231552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131171161 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚870763917874688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993814307840000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15585147762471 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157358136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57428287 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21651433763840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884353959455082e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131171161 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15608457765376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131171161 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚870763917874688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993814307840000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15585147762471 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157358136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57428287 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21651433763840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884353959455082e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131171161 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15608457765376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57427769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131171155 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1009425897390592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15611563159040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135931656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009422765605376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _runtime‚7374.300746948 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78503890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157358136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993814334231552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15608457765376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131171155 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157381242×\ªÙ2Ó- _wandbruntime‚7375  _runtime‚7375oxÕ'Ô˜N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288590803 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135949513 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚992463107963575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78514140 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21654400870912 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚993945828319744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288590803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78514143 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 % _timestamp‚1.7884353968652575e+09 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15587256555779 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15584153086647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993945801928192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157402358 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚993945828319744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131188439 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870879295845888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157402358 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15610570079232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288567665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15610570079232 _step‚600 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15613675486208 +ÊÕN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009559503805952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135950014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57435373 trainer/global_step‚7299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15610570079232 _step‚600 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15613675486208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288590803 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135949513 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚992463107963575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78514140 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21654400870912 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚993945828319744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288590803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78514143 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 % _timestamp‚1.7884353968652575e+09 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15587256555779 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15584153086647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993945801928192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157402358 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚993945828319744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131188439 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870879295845888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157402358 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15610570079232 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288567665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136827470 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15717193265664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132035355 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57811432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15714087872000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290451343 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1016103100121600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚998910034373159 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15687497539755 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚998906905476267 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21799190134272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚603 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1000389012249600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15690601008679 trainer/global_step‚7349 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132035355 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884354437577984e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚7422.113048192 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1000389038641152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1000389012249600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚876532189317120 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79016540 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚876532268271616 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290474455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136827972 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚983219407936512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158439094 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚854732999182848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158415988 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1000389038641152 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1016103100121600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚998910034373159 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15687497539755 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚998906905476267 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21799190134272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚603 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1000389012249600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15690601008679 trainer/global_step‚7349 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132035355 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884354437577984e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚7422.113048192 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1000389038641152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1000389012249600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚876532189317120 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79016540 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚876532268271616 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290474455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136827972 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚983219407936512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158439094 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚854732999182848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158415988 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1000389038641152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136827470 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15717193265664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132035355 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segùu«h%ment.large_pool.allocated‚1302Û7ü«Ã2å- _wandbruntime‚7423  _runtime‚7423x#"'æœN K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158437078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136845193 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290512855 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132052645 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚854845410046976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚999038411270583 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1016239838322176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15719305592832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1000520506337792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚983348830365696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290512855 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1016239838322176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290489717 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79026793 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1000520532729344 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15716200185856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158460210 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚7423.019977196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1000520532729344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57818899 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132052645 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚876647548669440 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚983348804937728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚604 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15719305592832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290489717 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136845692 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚876647627610624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15689606332855 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528j à¬''ç¡N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290489717 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136845692 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚876647627610624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15689606332855 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158437078 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136845193 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290512855 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132052645 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚854845410046976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚999038411270583 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1016239838322176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15719305592832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1000520506337792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚983348830365696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290512855 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1016239838322176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290489717 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79026793 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1000520532729344 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15716200185856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158460210 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚7423.019977196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1000520532729344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57818899 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132052645 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚876647548669440 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚983348804937728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚604 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15719305592832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878ܵç½2è- _wandbruntime‚7423  _runtime‚7423™ j^éØ /loss/FBt2/detection_loss‚1.7274549573659896 trainer/global_step‚7349 0loss/SRqqt3/detection_loss‚2.160458793016997 epoch‚6 _step‚605 2loss/FBt4/assignment_loss‚0.014664499503039805 /loss/FRt1/assignment_loss‚1.437769927183787 3loss/FBt2/assignment_loss‚0.0007592928582653257 1loss/FBt3/assignment_loss‚0.09366453285864244 /loss/FBt1/detection_loss‚0.8247167133218185 _runtime‚7423.020902483 2loss/SRqqt1/assignment_loss‚1.0023573816932263 0loss/SRqqt4/detection_loss‚2.515298057347536 .loss/FBt3/detection_loss‚2.625380507537297 /loss/FRt1/detection_loss‚1.4142629046738149 /loss/FRt2/detection_loss‚2.3289518102686455 0loss/FRt3/assignment_loss‚3.3836084288709305 2loss/SRqqt2/assignment_loss‚1.3802086072308677 1loss/SRqqt1/detection_loss‚1.0482714507887338 /loss/FRt3/detection_loss‚2.6204370102461647 1loss/SRqqt2/detection_loss‚2.2520896841068656 % _timestamp‚1.7884354446659062e+09 %loss/total_loss‚42.51636097799899 -loss/FRt2/assignment_loss‚3.1611456769578 /loss/FRt4/assignment_loss‚3.851108933196348 3loss/FBt1/assignment_loss‚0.0018149644119864145 2loss/SRqqt3/assignment_loss‚1.4464510829103263 2loss/SRqqt4/assignment_loss‚1.5130884842947125 .loss/FRt4/detection_loss‚2.890667140483856 .loss/FBt4/detection_loss‚2.821730136871338FVü«cêÝ 0loss/SRqqt3/detection_loss‚2.160458793016997 epoch‚6 _step‚605 2loss/FBt4/assignment_loss‚0.014664499503039805 /loss/FRt1/assignment_loss‚1.437769927183787 3loss/FBt2/assignment_loss‚0.0007592928582653257 1loss/FBt3/assignment_loss‚0.09366453285864244 /loss/FBt1/detection_loss‚0.8247167133218185 _runtime‚7423.020902483 2loss/SRqqt1/assignment_loss‚1.0023573816932263 0loss/SRqqt4/detection_loss‚2.515298057347536 .loss/FBt3/detection_loss‚2.625380507537297 /loss/FRt1/detection_loss‚1.4142629046738149 /loss/FRt2/detection_loss‚2.3289518102686455 0loss/FRt3/assignment_loss‚3.3836084288709305 2loss/SRqqt2/assignment_loss‚1.3802086072308677 1loss/SRqqt1/detection_loss‚1.0482714507887338 /loss/FRt3/detection_loss‚2.6204370102461647 1loss/SRqqt2/detection_loss‚2.2520896841068656 % _timestamp‚1.7884354446659062e+09 %loss/total_loss‚42.51636097799899 -loss/FRt2/assignment_loss‚3.1611456769578 /loss/FRt4/assignment_loss‚3.851108933196348 3loss/FBt1/assignment_loss‚0.0018149644119864145 2loss/SRqqt3/assignment_loss‚1.4464510829103263 2loss/SRqqt4/assignment_loss‚1.5130884842947125 .loss/FRt4/detection_loss‚2.890667140483856 .loss/FBt4/detection_loss‚2.821730136871338 /loss/FBt2/detection_loss‚1.7274549573659896 trainer/global_step‚7349Ý€srYžë‚(:“ ö·åÔ¾‰ˆµ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚140.915' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.182999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o7~kwì‚(:ì ö·åÔ†ÏÙ¶ disk.drbd2266.out‚0 disk.rbd1.out‚ 3.986328125( disk.nvme0c0n1.out‚61111.09326171875 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125$ memory_percent‚99.50134754180908 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚61111.09326171875 network.sent‚ 1.8152057e+07 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚19.03145314709857 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚61111.09326171875 disk.sda3.out‚ 279.1796875 disk.sda.out‚ 279.1796875' proc.memory.availableMB‚ 81.69921875* proc.memory.percent‚17.966747283935547 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 279.1796875 network.recv‚ 2.511075e+06% disk./.usageGB‚283.58240509033203 disk.drbd2265.out‚0 proc.cpu.threads‚17! proc.memory.rssMB‚ 2943.671875 cpu‚25.287181099337978 disk.dm-1.in‚0.203125 disk.sda1.out‚0 disk.sda.in‚0.203125:ãé5“í‚(:ˆ …¸åÔ÷ˆ…µ gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚296.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.2754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î͉Fyî‚(:î …¸åÔ´ˆÑ¶ proc.cpu.threads‚17 disk.nvme0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚61213.61279296875# disk.dm-0.out‚61213.61279296875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.826117e+07 network.recv‚ 2.518393e+06 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.sda3.out‚ 279.72265625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 279.72265625 disk.sda2.out‚0 disk.rbd1.out‚ 3.986328125 disk.dm-0.in‚ 466.90234375 disk.rbd1.in‚0 disk.sda.in‚0.203125# proc.memory.rssMB‚ 2944.29296875 cpu‚25.286705058661795& disk.nvme0n1.out‚61213.61279296875 disk.sda1.out‚0 disk.sda.out‚ 279.72265625 disk.drbd2265.out‚0& proc.memory.availableMB‚ 80.8671875) proc.memory.percent‚17.97053813934326$ disk./.usageGB‚283.6316146850586 disk.dm-1.in‚0.203125$ memory_percent‚99.50642585754395 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0* disk./.usagePercent‚19.034755644290268 disk.rbd0.out‚0 disk.sda2.in‚0€ù+ðžï‚(:“ ”¸å԰ضµ gpu.0.gpu‚89.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚285.575' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.114999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†>$‡{ð‚(:𠔸åÔ¬ŸÔ¶ disk.drbd2080.out‚0$ memory_percent‚99.50854778289795 cpu‚25.286230238949386 disk.sda2.in‚0 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚61293.24169921875# proc.memory.rssMB‚ 2945.02734375 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 280.09765625 disk.rbd0.in‚0& disk.nvme0n1.out‚61293.24169921875# disk.dm-0.out‚61293.24169921875 disk.drbd1716.in‚0 disk.drbd2265.out‚0* disk./.usagePercent‚19.036479599025842 disk.drbd2080.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2265.in‚0$ disk./.usageGB‚283.6573028564453 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.986328125" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 280.09765625 disk.sda1.out‚0) proc.memory.percent‚17.97502040863037 network.sent‚ 1.8266501e+07 disk.dm-1.out‚ 280.09765625 disk.dm-0.in‚ 466.90234375' proc.memory.availableMB‚ 80.51953125 proc.cpu.threads‚17 network.recv‚ 2.522037e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.out‚0 disk.drbd1716.out‚0Z²Ê2ñ- _wandbruntime‚7469  _runtime‚7469ß¾ëÂÇòÁ & lr-AdamW/pg2‚8.032327201709277e-05 trainer/global_step‚7399 % _timestamp‚1.7884354915946617e+09 _runtime‚7469.949714596 _step‚606 & lr-AdamW/pg1‚8.032327201709277e-05ÙÒþæÌóÆ _step‚606 & lr-AdamW/pg1‚8.032327201709277e-05 & lr-AdamW/pg2‚8.032327201709277e-05 trainer/global_step‚7399 % _timestamp‚1.7884354915946617e+09 _runtime‚7469.949714596ÞƇ2ô- _wandbruntime‚7469  _runtime‚7469é[Šm¾õ“P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788435491595445e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137721442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58192252 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15819717978624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79529190 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132899561 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15819717978624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58191736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚882299819312640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1005482208783275 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚989689257997312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚607 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15822823372288 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1022783434637824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1022783434637824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159473840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚860353570523648 ZLDeviceStatsMonityôxQ[or.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132899555 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159496946 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1006963743050752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21946327743488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132899561 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1006963743050752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚860353542389248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159496946 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚989689283425280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚7469.950727288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292396507 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15796054254887 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159473840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚882299898267136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15792950785963 trainer/global_step‚7399 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1006963716659200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1005485337680167 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1022786566423040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21946276923392yw(d(ö˜P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21946276923392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788435491595445e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137721442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58192252 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15819717978624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79529190 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132899561 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15819717978624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58191736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚882299819312640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1005482208783275 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚989689257997312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚607 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15822823372288 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1022783434637824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1022783434637824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159473840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚860353570523648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132899555 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159496946 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1006963743050752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21946327743488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132899561 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1006963743050752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚860353542389248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159496946 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚989689283425280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚7469.950727288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292396507 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15796054254887 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159473840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚882299898267136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15792950785963 trainer/global_step‚7399 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1006963716659200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1005485337680167 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1022786566423040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ß+úáÆœ÷‚(:‘ £¸åÔÈŽõµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚130.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.139999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,8÷°mø‚(:â £¸åÔǘ„¸ disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2945.96484375 cpu‚25.285781154301763 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.sda.out‚ 280.6953125 disk.drbd2266.out‚0$ memory_percent‚99.51705932617188 proc.cpu.threads‚17* disk./.usagePercent‚19.035583828663537$ disk.nvme0n1.out‚61369.564453125) proc.memory.percent‚17.98074245452881$ disk./.usageGB‚283.6439552307129 disk.drbd2265.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0" proc.memory.availableMB‚79.125 disk.drbd2080.out‚0! disk.dm-0.out‚61369.564453125 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 280.6953125 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 280.6953125& disk.nvme0c0n1.out‚61369.564453125 network.sent‚ 1.8272527e+07 network.recv‚ 2.526621e+06 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125Ë”ôl2ù- _wandbruntime‚7470  _runtime‚7470óà]}#'úN OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132916839 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159494930 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137739182 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1007095210747392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159494930 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚860465953253376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292434907 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21949212030464 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79539443 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788435492511647e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1022920172838400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚608 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1022917041039872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132916839 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚7470.867659349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1005613714577591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15821830292480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15798163048195 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137738672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15821830292480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292411769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21949262837248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292434907 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _step‚608 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1022917041039872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132916839 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚7470.867659349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1005613714577591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15821830292480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15798163048195 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137738672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15821830292480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292411769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21949262837248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292434907 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1007095210747392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159494930 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚860465953253376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292434907 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624àW _³2ü- _wandbruntime‚7470  _runtime‚7470)´naýÛ 2loss/SRqqt4/assignment_loss‚0.9927761353552341 .loss/FRt3/detection_loss‚2.433936568952742 .loss/FRt4/detection_loss‚2.804284630343318 1loss/SRqqt4/detection_loss‚2.4837862143913907 /loss/FBt2/detection_loss‚2.4761230995257697 .loss/FBt3/detection_loss‚2.903044973100935 %loss/total_loss‚42.17297754121994 0loss/FRt3/assignment_loss‚2.9286143623647236 /loss/FRt1/detection_loss‚1.4765764900980747 .loss/FRt2/detection_loss‚2.383047972854815 /loss/FBt4/detection_loss‚2.3774570636451244 2loss/SRqqt2/assignment_loss‚1.5785553892227737 1loss/FBt1/assignment_loss‚0.05934141004154285 /loss/FBt1/detection_loss‚0.8662218331758466 trainer/global_step‚7399 0loss/FRt1/assignment_loss‚1.7734205469693223 /loss/FRt2/assignment_loss‚3.699949556275418 2loss/SRqqt1/assignment_loss‚0.7595834468957037 1loss/FBt2/assignment_loss‚0.09697400471122819 1loss/FBt4/assignment_loss‚0.07149316446155751 1loss/SRqqt1/detection_loss‚0.9461592142935842 1loss/SRqqt3/detection_loss‚2.1036238763067456 _runtime‚7470.868691978 % _timestamp‚1.7884354925137098e+09 _step‚609 0loss/FRt4/assignment_loss‚2.9156987592577934 2loss/SRqqt3/assignment_loss‚1.5371536028260986 4loss/FBt3/assignment_loss‚0.00040541509331656504 1loss/SRqqt2/detection_loss‚2.5047498110568887 epoch‚6ƒ=|.fþà epoch‚6 2loss/SRqqt4/assignment_loss‚0.9927761353552341 .loss/FRt3/detection_loss‚2.433936568952742 .loss/FRt4/detection_loss‚2.804284630343318 1loss/SRqqt4/detection_loss‚2.4837862143913907 /loss/FBt2/detection_loss‚2.4761230995257697 .loss/FBt3/detection_loss‚2.903044973100935 %loss/total_loss‚42.17297754121994 0loss/FRt3/assignment_loss‚2.9286143623647236 /loss/FRt1/detection_loss‚1.4765764900980747 .loss/FRt2/detection_loss‚2.383047972854815 /loss/FBt4/detection_loss‚2.3774570636451244 2loss/SRqqt2/assignment_loss‚1.5785553892227737 1loss/FBt1/assignment_loss‚0.05934141004154285 /loss/FBt1/detection_loss‚0.8662218331758466 trainer/global_step‚7399 0loss/FRt1/assignment_loss‚1.7734205469693223 /loss/FRt2/assignment_loss‚3.699949556275418 2loss/SRqqt1/assignment_loss‚0.7595834468957037 1loss/FBt2/assignment_loss‚0.09697400471122819 1loss/FBt4/assignment_loss‚0.07149316446155751 1loss/SRqqt1/detection_loss‚0.9461592142935842 1loss/SRqqt3/detection_loss‚2.1036238763067456 _runtime‚7470.868691978 % _timestamp‚1.7884354925137098e+09 _step‚609 0loss/FRt4/assignment_loss‚2.9156987592577934 2loss/SRqqt3/assignment_loss‚1.5371536028260986 4loss/FBt3/assignment_loss‚0.00040541509331656504 1loss/SRqqt2/detection_loss‚2.5047498110568887áy±\Ê“ÿ‚(:ˆ ²¸åԸ긵 gpu.0.gpu‚69.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚281.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷pEw€ ‚(:ì ²¸åÔÎõ¿» disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125) disk./.usagePercent‚19.03794780223471 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 281.16796875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2080.out‚0$ memory_percent‚99.51779842376709' proc.memory.availableMB‚ 79.00390625$ disk./.usageGB‚283.6791801452637 disk.drbd2266.in‚0 disk.sda3.out‚ 281.16796875# proc.memory.rssMB‚ 2946.45703125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 281.16796875 disk.drbd1716.out‚0) proc.memory.percent‚17.98374652862549 cpu‚25.285252562195268 disk.drbd2266.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚61449.0361328125" disk.dm-0.out‚61449.0361328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 3.986328125 disk.drbd1716.in‚0 network.recv‚ 2.533029e+06% disk.nvme0n1.out‚61449.0361328125 disk.sda1.out‚0 network.sent‚ 1.8369735e+07 disk.drbd2265.in‚0༠‚(:’ Á¸åÔÔ¹ñ´ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚132.06' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.412000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰¼DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1029463769154048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58574008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15925348085248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133763761 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160554798 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80041840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133763761 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7449 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1029466900939264 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1013538447460352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133763755 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1012060640987175 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22093858799104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚865974085595648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1013538447460352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138615845 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1029463769154048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138616358 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133763755 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15928453478912 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22093909619200 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294295447 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15928453478912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6ÉVñ’$%Š —P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1029466900939264 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1013538447460352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133763755 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1012060640987175 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22093858799104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚865974085595648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1013538447460352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138615845 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1029463769154048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138616358 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133763755 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15928453478912 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22093909619200 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294295447 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15928453478912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1029463769154048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58574008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15925348085248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133763761 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160554798 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80041840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133763761 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7449 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58581752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚996288505059328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1013669915156992 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138634339 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80052093 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1013669941548544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133781039 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1013669941548544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133781045 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1029597375556096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888183397722624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1012189017884599 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15903616294403 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294356959 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138633842 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22096824418816 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15930565806080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160552782 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133781045 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76844032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22096873128448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884355399932141e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1013669915156992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294333821 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚996288530487296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294356959 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294333821 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884355399932141e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1013669915156992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294333821 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚996288530487296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294356959 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294333821 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58581752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚996288505059328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1013669915156992 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138634339 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80052093 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1013669941548544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133781039 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1013669941548544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133781045 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1029597375556096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888183397722624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1012189017884599 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15903616294403 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294356959 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138633842 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22096824418816 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15930565806080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160552782 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ä… 2Ž - _wandbruntime‚7518  _runtime‚7518d9€‚_ Ù /loss/FRt2/assignment_loss‚3.263704006870588 2loss/SRqqt1/assignment_loss‚1.0110053580069422 0loss/FBt4/assignment_loss‚0.1226496408094785 .loss/FRt2/detection_loss‚2.314014058560133 _runtime‚7518.350472414 0loss/FRt3/assignment_loss‚2.4287630915641785 1loss/SRqqt4/assignment_loss‚2.003173230588436 /loss/FRt3/detection_loss‚2.7082286333215646 /loss/FBt1/detection_loss‚0.6872426764214341 epoch‚6 % _timestamp‚1.7884355399954517e+09 _step‚613 2loss/SRqqt2/assignment_loss‚1.7911036870545811 1loss/SRqqt2/detection_loss‚2.0384969426525963 1loss/SRqqt4/detection_loss‚2.5849913279215495 .loss/FBt3/detection_loss‚2.222541218996048 trainer/global_step‚7449 1loss/SRqqt1/detection_loss‚0.9663206930984469 0loss/FRt4/assignment_loss‚3.9977308948834738 /loss/FBt2/detection_loss‚2.8332892060279846 &loss/total_loss‚42.807652724384255 /loss/FBt4/detection_loss‚1.9443143627473287 0loss/FRt1/assignment_loss‚1.6361605713764826 2loss/SRqqt3/assignment_loss‚1.4701993582770227 1loss/FBt1/assignment_loss‚0.03803732834629159 1loss/FBt3/assignment_loss‚0.25614777256269006 /loss/FRt1/detection_loss‚1.5838773319497705 /loss/FRt4/detection_loss‚2.8283126930395763 1loss/FBt2/assignment_loss‚0.17165323728148416 1loss/SRqqt3/detection_loss‚1.9056954020261765n¾´Œd Þ 2loss/SRqqt1/assignment_loss‚1.0110053580069422 0loss/FBt4/assignment_loss‚0.1226496408094785 .loss/FRt2/detection_loss‚2.314014058560133 _runtime‚7518.350472414 0loss/FRt3/assignment_loss‚2.4287630915641785 1loss/SRqqt4/assignment_loss‚2.003173230588436 /loss/FRt3/detection_loss‚2.7082286333215646 /loss/FBt1/detection_loss‚0.6872426764214341 epoch‚6 % _timestamp‚1.7884355399954517e+09 _step‚613 2loss/SRqqt2/assignment_loss‚1.7911036870545811 1loss/SRqqt2/detection_loss‚2.0384969426525963 1loss/SRqqt4/detection_loss‚2.5849913279215495 .loss/FBt3/detection_loss‚2.222541218996048 trainer/global_step‚7449 1loss/SRqqt1/detection_loss‚0.9663206930984469 0loss/FRt4/assignment_loss‚3.9977308948834738 /loss/FBt2/detection_loss‚2.8332892060279846 &loss/total_loss‚42.807652724384255 /loss/FBt4/detection_loss‚1.9443143627473287 0loss/FRt1/assignment_loss‚1.6361605713764826 2loss/SRqqt3/assignment_loss‚1.4701993582770227 1loss/FBt1/assignment_loss‚0.03803732834629159 1loss/FBt3/assignment_loss‚0.25614777256269006 /loss/FRt1/detection_loss‚1.5838773319497705 /loss/FRt4/detection_loss‚2.8283126930395763 1loss/FBt2/assignment_loss‚0.17165323728148416 1loss/SRqqt3/detection_loss‚1.9056954020261765 /loss/FRt2/assignment_loss‚3.263704006870588åͺYÔž‘ ‚(:“ ߸åÔÿ†°µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚125.397' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.079400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HPQ‹r’ ‚(:ç ߸åÔüŽ¡· network.recv‚ 2.547681e+06 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125$ proc.memory.availableMB‚77.46875 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.986328125" disk.dm-0.out‚61703.1884765625 proc.memory.rssMB‚ 2947.96875) proc.memory.percent‚17.99297332763672 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 282.76953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 282.76953125 disk.rbd0.in‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 282.76953125$ memory_percent‚99.52716827392578 cpu‚25.283881233917057 proc.cpu.threads‚17 network.sent‚ 1.8479591e+07* disk./.usagePercent‚19.044471386222224 disk.nvme0n1.in‚ 466.90234375% disk.nvme0n1.out‚61703.1884765625' disk.nvme0c0n1.out‚61703.1884765625 disk.sda.in‚0.203125$ disk./.usageGB‚283.7763862609863 disk.dm-0.in‚ 466.90234375 disk.drbd1716.in‚0!UÖ•““ ‚(:ˆ î¸åÔ¨«²¶ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.003' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃØ1Mx” ‚(:í î¸åÔÙß²¸ disk.sda.in‚0.203125% disk.nvme0n1.out‚61740.1611328125 disk.rbd1.in‚0% disk./.usageGB‚283.80064392089844 disk.sda1.in‚0 disk.sda.out‚ 283.26171875 disk.rbd0.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.in‚0.203125" proc.memory.rssMB‚ 2948.3046875 disk.nvme0n1.in‚ 466.90234375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚61740.1611328125 disk.sda3.out‚ 283.26171875$ memory_percent‚99.52921867370605 cpu‚25.283409782495447 proc.cpu.threads‚17 network.sent‚ 1.8588551e+07 network.recv‚ 2.554085e+06* disk./.usagePercent‚19.046099338132457 disk.drbd2080.in‚0 disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚61740.1611328125* proc.memory.percent‚17.995023727416992 disk.drbd2265.in‚0 disk.dm-1.out‚ 283.26171875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 77.1328125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.986328125 disk.drbd1716.in‚0DL.6• ‚(:’ ý¸åÔ܆µ gpu.0.gpu‚88.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.78' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.955999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ziy– ‚(:î ý¸åÔÏîĶ disk.rbd0.in‚0 disk.rbd1.out‚ 3.986328125$ memory_percent‚99.52938556671143 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0' proc.memory.availableMB‚ 77.10546875# proc.memory.rssMB‚ 2948.33203125 cpu‚25.282961238626033 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.in‚0# disk.dm-0.out‚61823.54638671875 network.recv‚ 2.558769e+06% disk./.usageGB‚283.83453369140625 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚61823.54638671875 disk.dm-0.in‚ 466.90234375 network.sent‚ 1.8594749e+07 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.dm-1.out‚ 283.9609375* proc.memory.percent‚17.995190620422363 disk.drbd2266.out‚0) disk./.usagePercent‚19.04837370906665 disk.sda.in‚0.203125 disk.sda3.in‚0.203125 disk.sda.out‚ 283.9609375 disk.rbd0.out‚0& disk.nvme0n1.out‚61823.54638671875 disk.sda3.out‚ 283.9609375zïfF2— - _wandbruntime‚7565  _runtime‚7565-ò€Å˜ ¿ % lr-AdamW/pg1‚7.74484956839216e-05 % lr-AdamW/pg2‚7.74484956839216e-05 trainer/global_step‚7499 % _timestamp‚1.7884355868521194e+09 _runtime‚7565.207165079 _step‚614-1•Ê™ Ä trainer/global_step‚7499 % _timestamp‚1.7884355868521194e+09 _runtime‚7565.207165079 _step‚614 % lr-AdamW/pg1‚7.74484956839216e-05 % lr-AdamW/pg2‚7.74484956839216e-05æø*82š - _wandbruntime‚7565  _runtime‚7565»³F7ñ› –P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22241803576832 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1018632815397291 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134627955 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16003857278379 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296217499 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1018635944294183 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134627961 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58957087 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1036144103670272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80554490 G>DeviceStatsMonitor.on_train_b¯z®+atch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16034083585536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139511067 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1020113125478400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16034083585536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161612650 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80554487 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7565.208271107 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1036147235455488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134627961 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚893836432378880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884355868529565e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161589544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1036144103670272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1002628958118912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16030978191872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1036147235455488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚871594628802048rõÎç!(œ ›P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1036147235455488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134627961 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚893836432378880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884355868529565e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161589544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1036144103670272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1002628958118912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16030978191872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1036147235455488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚871594628802048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22241803576832 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1018632815397291 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134627955 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16003857278379 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296217499 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1018635944294183 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134627961 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58957087 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1036144103670272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80554490 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16034083585536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139511067 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1020113125478400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16034083585536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161612650 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80554487 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7565.208271107çsÆà2 - _wandbruntime‚7566  _runtime‚7566dâ#é%'ž ŸN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161610634 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7566.13015103 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58964255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893951857200640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚871707039666176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚7499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134645245 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134645239 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1002758355120128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139528995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296279011 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296255873 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16005966071479 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296255873 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134645245 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16009069540611 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884355877741368e+09 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1018767450088707 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1036280841870848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161610634 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1020244645958144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80564743 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139529486 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1002758380548096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16033090505728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16033090505728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22244738593280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1020244619566592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58964743 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161633766 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893951778259456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1036280841870848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1020244619566592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80564740 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16036195912704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296279011 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1018764321191607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22244789400064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1020244645958144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚871707067800576 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161633766 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16036195912704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚491 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976>Š{^:Ÿ ¤N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58964255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893951857200640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚871707039666176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚7499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134645245 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134645239 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1002758355120128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139528995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296279011 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296255873 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16005966071479 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296255873 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134645245 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16009069540611 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884355877741368e+09 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1018767450088707 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1036280841870848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161610634 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1020244645958144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80564743 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139529486 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1002758380548096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16033090505728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16033090505728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22244738593280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1020244619566592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58964743 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161633766 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893951778259456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 MGDeviceStatsMonitor.on_train_ÑU»Ãðbatch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1036280841870848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1020244619566592 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80564740 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16036195912704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296279011 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1018764321191607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22244789400064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1020244645958144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚871707067800576 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161633766 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16036195912704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚491 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161610634 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7566.13015103 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096è7P2  - _wandbruntime‚7566  _runtime‚7566; -a]¡ × trainer/global_step‚7499 0loss/FBt3/assignment_loss‚0.5138054741313681 /loss/FBt3/detection_loss‚2.3019253313541412 _step‚617 /loss/FRt4/assignment_loss‚3.781786235049367 1loss/SRqqt2/assignment_loss‚1.462550370891889 2loss/SRqqt4/assignment_loss‚1.5927758154769738 3loss/FBt4/assignment_loss‚0.0031639752404381788 0loss/SRqqt4/detection_loss‚2.699212705095609 epoch‚6 _runtime‚ 7566.13150717 .loss/FRt2/assignment_loss‚3.65404030969066 0loss/FRt3/assignment_loss‚3.2048971779418713 2loss/SRqqt1/assignment_loss‚0.8004338981376754 1loss/SRqqt2/detection_loss‚2.3352693061033887 /loss/FBt2/detection_loss‚2.2493614703416824 1loss/FBt2/assignment_loss‚0.03381831869288968 /loss/FRt3/detection_loss‚2.6618546924807807 1loss/SRqqt1/detection_loss‚0.8877993014491634 1loss/SRqqt3/detection_loss‚1.6479717626103334 %loss/total_loss‚43.10962220566688 0loss/FRt1/assignment_loss‚1.6133689672657938 /loss/FRt1/detection_loss‚1.7200334511245743 /loss/FRt4/detection_loss‚2.8768874183297157 .loss/FBt4/detection_loss‚2.443894531045641 % _timestamp‚1.7884355877764974e+09 2loss/SRqqt3/assignment_loss‚1.3366203463769384 .loss/FBt1/detection_loss‚0.780435907103541 2loss/FBt1/assignment_loss‚0.009560684170857514 /loss/FRt2/detection_loss‚2.4981547555615826 wCb¢ Ü 2loss/FBt1/assignment_loss‚0.009560684170857514 /loss/FRt2/detection_loss‚2.4981547555615826 trainer/global_step‚7499 0loss/FBt3/assignment_loss‚0.5138054741313681 /loss/FBt3/detection_loss‚2.3019253313541412 _step‚617 /loss/FRt4/assignment_loss‚3.781786235049367 1loss/SRqqt2/assignment_loss‚1.462550370891889 2loss/SRqqt4/assignment_loss‚1.5927758154769738 3loss/FBt4/assignment_loss‚0.0031639752404381788 0loss/SRqqt4/detection_loss‚2.699212705095609 epoch‚6 _runtime‚ 7566.13150717 .loss/FRt2/assignment_loss‚3.65404030969066 0loss/FRt3/assignment_loss‚3.2048971779418713 2loss/SRqqt1/assignment_loss‚0.8004338981376754 1loss/SRqqt2/detection_loss‚2.3352693061033887 /loss/FBt2/detection_loss‚2.2493614703416824 1loss/FBt2/assignment_loss‚0.03381831869288968 /loss/FRt3/detection_loss‚2.6618546924807807 1loss/SRqqt1/detection_loss‚0.8877993014491634 1loss/SRqqt3/detection_loss‚1.6479717626103334 %loss/total_loss‚43.10962220566688 0loss/FRt1/assignment_loss‚1.6133689672657938 /loss/FRt1/detection_loss‚1.7200334511245743 /loss/FRt4/detection_loss‚2.8768874183297157 .loss/FBt4/detection_loss‚2.443894531045641 % _timestamp‚1.7884355877764974e+09 2loss/SRqqt3/assignment_loss‚1.3366203463769384 .loss/FBt1/detection_loss‚0.780435907103541é¿.‰z£ ‚(:’ Œ¹åÔòªžµ gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.78' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.955999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b‚Õ4v¤ ‚(:ë Œ¹åÔåĵ¸ disk.drbd2265.out‚0) proc.memory.percent‚17.99781322479248 cpu‚25.28252128037968& disk.nvme0n1.out‚61921.14208984375& proc.memory.availableMB‚ 76.3671875$ disk./.usageGB‚283.8675842285156 disk.sda3.out‚ 284.3515625 disk.drbd1716.in‚0 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 3.986328125( disk.nvme0c0n1.out‚61921.14208984375 disk.dm-0.in‚ 466.90234375 network.recv‚ 2.562509e+06 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0 network.sent‚ 1.8613217e+07 disk.drbd1716.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375$ memory_percent‚99.53389167785645# disk.dm-0.out‚61921.14208984375 disk.sda.out‚ 284.3515625 disk.drbd2265.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2948.76171875* disk./.usagePercent‚19.050591758343305 disk.dm-1.out‚ 284.3515625 disk.drbd2266.in‚0 disk.drbd2080.out‚0[Ö‰“¥ ‚(:ˆ ›¹åÔâœã´ gpu.0.gpu‚82.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚132.213' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.4426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 k«em¦ ‚(:â ›¹åԻ÷ disk.dm-1.in‚0.203125' proc.memory.availableMB‚ 76.41796875 proc.memory.rssMB‚2949 cpu‚25.282036813389688 network.sent‚ 1.8709028e+07" disk.nvme0c0n1.in‚ 466.90234375 disk.sda.in‚0.203125 disk.sda.out‚ 284.8984375' proc.memory.percent‚17.999267578125 network.recv‚ 2.568852e+06$ disk./.usageGB‚283.8556785583496 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 284.8984375 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0$ memory_percent‚99.53358173370361* disk./.usagePercent‚19.049792758828943 disk.rbd0.in‚0( disk.nvme0c0n1.out‚62015.25634765625 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚62015.25634765625 disk.sda2.in‚0 disk.rbd1.out‚ 3.986328125 disk.drbd2080.in‚0 disk.drbd2266.out‚0# disk.dm-0.out‚62015.25634765625 disk.sda3.in‚0.203125 disk.sda3.out‚ 284.8984375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0Œ²¨§ ‚(:’ ª¹åÔ–º›µ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.004200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Ç#™[¨ ‚(:Ð ª¹åÔµ˜ƒ·" disk.nvme0c0n1.out‚ 62089.71875) disk./.usagePercent‚19.05204997725834 disk.dm-0.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 3.986328125 proc.memory.rssMB‚ 2949.09375$ memory_percent‚99.53594207763672 network.recv‚ 2.57279e+06 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 285.515625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 285.515625 disk.rbd0.in‚0 disk.nvme0n1.out‚ 62089.71875 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2265.in‚0$ proc.memory.availableMB‚76.03125* proc.memory.percent‚17.999839782714844 cpu‚25.28158403669563 proc.cpu.threads‚17 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-0.out‚ 62089.71875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 network.sent‚ 1.8715029e+07$ disk./.usageGB‚283.8893127441406 disk.sda3.out‚ 285.515625 disk.rbd0.out‚0 disk.rbd1.in‚0ü>ºb2© - _wandbruntime‚7612  _runtime‚7612måmƪ À $ _timestamp‚1.788435634625357e+09 _runtime‚7612.980386289 _step‚618 & lr-AdamW/pg1‚7.601872189873853e-05 & lr-AdamW/pg2‚7.601872189873853e-05 trainer/global_step‚7549SûÒñË« Å & lr-AdamW/pg1‚7.601872189873853e-05 & lr-AdamW/pg2‚7.601872189873853e-05 trainer/global_step‚7549 $ _timestamp‚1.788435634625357e+09 _runtime‚7612.980386289 _step‚618ê`§½2¬ - _wandbruntime‚7612  _runtime‚7612ŠéJ(­ •P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1042824438186496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚877215200142848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162647396 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81067137 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1009098808179712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22389159513088 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135492161 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16109310524587 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16139713692160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1026687856279552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1025211247601191 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1042827569971712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1042824438186496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1026687829888000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22389106595840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135492155 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1042827569971712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140407338 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _runtime‚ 7612.98133513 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16139713692160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1026687856279552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16136608298496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162670502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚899604359655936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135492155 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚899604278604288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59340713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16112413993511 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298162663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298139551 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162670502 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140407853 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884356346260736e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298139551 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162670502 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140407853 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884356346260736e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1026687856279552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1025211247601191 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1042827569971712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1042824438186496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1026687829888000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22389106595840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135492155 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1042827569971712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140407338 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _runtime‚ 7612.98133513 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16139713692160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1026687856279552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16136608298496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162670502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚899604359655936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135492155 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚899604278604288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59340713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16112413993511 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298162663ëÄÛèå2¯ - _wandbruntime‚7613  _runtime‚7613»›d4° žN RBDeviceStatsMonitor.on_train_batch_end/rese`Djð&rved_bytes.all.allocated‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚7613.887521573 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22392126482432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16141826019328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877327611006976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1042961176387072 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16111419317687 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877327582872576 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22392073578496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140425179 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1025339624498615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 # _timestamp‚1.78843563553148e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1026819350367744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16138720612352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298201063 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298201063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135509445 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81077393 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1042961176387072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1009228205180928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚899719656451072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1009228230608896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135509439 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚7549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1025342753395715 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298177925 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1042958044588544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162691618 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140425677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚899719737489408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162691618 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1026819350367744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16138720612352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135509439 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59348284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59347789 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 _step‚620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162668486 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162668486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 _step‚620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162668486 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162668486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚7613.887521573 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22392126482432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16141826019328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877327611006976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1042961176387072 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16111419317687 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877327582872576 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22392073578496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140425179 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1025339624498615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 # _timestamp‚1.78843563553148e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1026819350367744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16138720612352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298201063 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298201063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135509445 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81077393 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1042961176387072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1009228205180928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚899719656451072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1009228230608896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135509439 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚7549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1025342753395715 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298177925 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1042958044588544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162691618 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140425677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚899719737489408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162691618 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1026819350367744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16138720612352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135509439ì;ó2² - _wandbruntime‚7613  _runtime‚7613ÐdÈÀZ³ Ô _runtime‚7613.888782423 0loss/FBt3/assignment_loss‚0.1813442097106114 0loss/FRt1/assignment_loss‚1.3447397767551361 0loss/FRt2/assignment_loss‚3.4679849909411535 0loss/SRqqt4/assignment_loss‚1.41251684778503 /loss/FRt3/detection_loss‚2.5680203766394882 % _timestamp‚1.7884356355337825e+09 .loss/FBt4/detection_loss‚2.374523917833964 _step‚621 /loss/FRt4/assignment_loss‚4.251738486466585 1loss/FBt2/assignment_loss‚0.07539858524008498 1loss/SRqqt2/detection_loss‚2.2323942342225243 2loss/SRqqt1/assignment_loss‚1.1087530843961315 2loss/SRqqt3/assignment_loss‚1.4919824051981172 1loss/SRqqt3/detection_loss‚1.5489469073712825 epoch‚6 /loss/FRt1/detection_loss‚1.4691265374422073 2loss/SRqqt2/assignment_loss‚1.3463810840073753 2loss/FBt1/assignment_loss‚0.013230250257133326 1loss/SRqqt1/detection_loss‚1.0170090382393353 /loss/FBt2/detection_loss‚2.8247992197672525 /loss/FBt3/detection_loss‚2.4412087003389993 .loss/FRt4/detection_loss‚2.733976236096135 0loss/FBt4/assignment_loss‚0.2788730694188012 -loss/FRt2/detection_loss‚2.42009076807234 /loss/FRt3/assignment_loss‚2.694050046113821 %loss/total_loss‚42.28616849820583 trainer/global_step‚7549 1loss/SRqqt4/detection_loss‚2.3479895708816394 /loss/FBt1/detection_loss‚0.6410901550106801R)ä_´ Ù /loss/FRt4/assignment_loss‚4.251738486466585 1loss/FBt2/assignment_loss‚0.07539858524008498 1loss/SRqqt2/detection_loss‚2.2323942342225243 2loss/SRqqt1/assignment_loss‚1.1087530843961315 2loss/SRqqt3/assignment_loss‚1.4919824051981172 1loss/SRqqt3/detection_loss‚1.5489469073712825 epoch‚6 /loss/FRt1/detection_loss‚1.4691265374422073 2loss/SRqqt2/assignment_loss‚1.3463810840073753 2loss/FBt1/assignment_loss‚0.013230250257133326 1loss/SRqqt1/detection_loss‚1.0170090382393353 /loss/FBt2/detection_loss‚2.8247992197672525 /loss/FBt3/detection_loss‚2.4412087003389993 .loss/FRt4/detection_loss‚2.733976236096135 0loss/FBt4/assignment_loss‚0.2788730694188012 -loss/FRt2/detection_loss‚2.42009076807234 /loss/FRt3/assignment_loss‚2.694050046113821 %loss/total_loss‚42.28616849820583 trainer/global_step‚7549 1loss/SRqqt4/detection_loss‚2.3479895708816394 /loss/FBt1/detection_loss‚0.6410901550106801 _runtime‚7613.888782423 0loss/FBt3/assignment_loss‚0.1813442097106114 0loss/FRt1/assignment_loss‚1.3447397767551361 0loss/FRt2/assignment_loss‚3.4679849909411535 0loss/SRqqt4/assignment_loss‚1.41251684778503 /loss/FRt3/detection_loss‚2.5680203766394882 % _timestamp‚1.7884356355337825e+09 .loss/FBt4/detection_loss‚2.374523917833964 _step‚621íA€x“µ ‚(:ˆ ¹¹åÔÏ↵ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚292.018' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ3Õ©y¶ ‚(:î ¹¹åÔ—§· disk.rbd1.out‚ 3.986328125 disk.drbd1716.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 286.11328125' proc.memory.availableMB‚ 75.37890625 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2080.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2949.51953125 proc.cpu.threads‚17# disk./.usageGB‚283.931941986084 disk.rbd0.out‚0( disk.nvme0c0n1.out‚62179.16748046875# disk.dm-0.out‚62179.16748046875 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 286.11328125 network.sent‚ 1.8721364e+07 disk.nvme0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 286.11328125$ memory_percent‚99.53992366790771 cpu‚25.28112783546322 disk.rbd0.in‚0 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.sda2.in‚0) proc.memory.percent‚18.00243854522705* disk./.usagePercent‚19.054910861453475 disk.drbd2265.in‚0 network.recv‚ 2.577706e+06& disk.nvme0n1.out‚62179.16748046875 disk.sda2.out‚0S:Oª“· ‚(:ˆ ȹåÔº§“µ gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û¶JLt¸ ‚(:é ȹåÔãñ÷ network.recv‚ 2.58352e+06 disk.drbd2080.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚283.9441833496094 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚62255.17529296875 disk.sda2.out‚0 disk.drbd2266.out‚0) disk./.usagePercent‚19.05573238963085 disk.drbd2265.in‚0 disk.rbd0.out‚0 network.sent‚ 1.8829503e+07 disk.sda.in‚0.203125 disk.sda.out‚ 286.640625 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375# disk.dm-0.out‚62255.17529296875' proc.memory.availableMB‚ 75.53515625* proc.memory.percent‚18.004202842712402 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 286.640625 disk.drbd1716.out‚0 cpu‚25.280673740371718 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚62255.17529296875 disk.dm-0.in‚ 466.90234375$ memory_percent‚99.53896999359131 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 286.640625 disk.sda1.out‚0# proc.memory.rssMB‚ 2949.80859375¤ Ö ž¹ ‚(:“ ×¹åÔõì­µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚299.276' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.855199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žxŽ`º ‚(:Õ ×¹åÔÞ“î¶ disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.90234375& proc.memory.percent‚18.00537109375 network.recv‚ 2.58722e+06 disk.sda3.in‚0.203125 disk.dm-1.out‚ 287.23828125 disk.drbd2080.out‚0 disk.drbd2265.out‚0$ proc.memory.availableMB‚75.34375 disk.drbd2080.in‚0 disk.sda.out‚ 287.23828125 disk.dm-1.in‚0.203125) disk./.usagePercent‚19.05771030384039 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125% disk.nvme0c0n1.out‚62351.43359375 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.280227494725494 network.sent‚ 1.8834545e+07$ disk./.usageGB‚283.9736557006836 disk.rbd0.out‚0# disk.nvme0n1.out‚62351.43359375 disk.drbd2266.in‚0# memory_percent‚99.5401382446289 proc.cpu.threads‚17 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 287.23828125 disk.drbd1716.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375 proc.memory.rssMB‚2950 disk.dm-0.out‚62351.43359375Ù½¢=2» - _wandbruntime‚7660  _runtime‚7660C‚bƼ À $ _timestamp‚1.788435682241199e+09 _runtime‚7660.596260036 _step‚622 & lr-AdamW/pg1‚7.459442201099467e-05 & lr-AdamW/pg2‚7.459442201099467e-05 trainer/global_step‚7599KfȆ˽ Å & lr-AdamW/pg2‚7.459442201099467e-05 trainer/global_step‚7599 $ _timestamp‚1.788435682241199e+09 _runtime‚7660.596260036 _step‚622 & lr-AdamW/pg1‚7.459442201099467e-05îE$ â2¾ - _wandbruntime‚7660  _runtime‚7660P—ÅR†¿ •P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1033262560689152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59722571 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1049507904487936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136356361 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16245343798784 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚7599 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300061603 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136356355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22536019085824 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1031783422011307 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136356355 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1033262534297600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1049504772702720 M=DeviceStatsMon^©û•itor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16242238405120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16217867239719 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81579790 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7660.597305349 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163728354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16214763770795 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163705248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1033262560689152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1015568683668480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1015568658240512 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300084715 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905371734300672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163728354 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141302358 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1049507904487936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚882835743349248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16245343798784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚882835715214848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1031786550908199Œj»æ (À šP H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1015568683668480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1015568658240512 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300084715 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905371734300672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163728354 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141302358 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1049507904487936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚882835743349248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16245343798784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚882835715214848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1031786550908199 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1033262560689152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59722571 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1049507904487936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136356361 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16245343798784 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚7599 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300061603 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136356355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22536019085824 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1031783422011307 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136356355 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1033262534297600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1049504772702720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16242238405120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16217867239719 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81579790 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7660.597305349 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163728354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16214763770795 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163705248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1033262560689152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6ïš962Á - _wandbruntime‚7661  _runtime‚7661Aóœt&'  N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141320131 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1049641510903296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1031914927805623 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300123115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59730091 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163726338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59730589 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300099977 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16244350718976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81590043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16219976033027 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22539027887616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884356831620653e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 trainer/global_step‚7599 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882948126078976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1015698055241728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300123115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81590040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1015698080669696 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136373645 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1049638379104768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136373639 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚905487182100992 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚624 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1049641510903296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16244350718976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16247456125952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141320632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1033394028385792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882948154213376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1033394028385792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22538974983680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1031918056702723 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163749470 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163726338 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1033394054777344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141320131 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1049641510903296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1031914927805623 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300123115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59730091 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163726338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59730589 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300099977 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16244350718976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81590043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16219976033027 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22539027887616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884356831620653e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 trainer/global_step‚7599 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882948126078976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1015698055241728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300123115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81590040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1015698080669696 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136373645 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1049638379104768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136373639 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚905487182100992 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚624 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1049641510903296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16244350718976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16247456125952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141320632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1033394028385792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882948154213376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1033394028385792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22538974983680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1031918056702723 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163749470 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163726338 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1033394054777344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096ð’é2Ä - _wandbruntime‚7661  _runtime‚7661?U>ŠXÅ Ò .loss/FRt3/detection_loss‚2.516434941742871 /loss/FRt4/detection_loss‚2.8746807306043562 1loss/SRqqt4/detection_loss‚2.1762348930040996 % _timestamp‚1.7884356831641965e+09 .loss/FRt2/detection_loss‚2.286036077667685 &loss/total_loss‚41.779014314551475 epoch‚6 /loss/FRt4/assignment_loss‚3.869626875846617 2loss/SRqqt1/assignment_loss‚0.9833274630408239 1loss/SRqqt3/assignment_loss‚1.633723319891621 2loss/SRqqt4/assignment_loss‚1.5622063751022022 /loss/FRt3/assignment_loss‚2.902262400936436 0loss/SRqqt1/detection_loss‚0.986666855272613 0loss/SRqqt2/detection_loss‚2.133634048539239 /loss/FBt2/detection_loss‚2.2291466858651905 _runtime‚7661.519169093 _step‚625 0loss/FRt1/assignment_loss‚1.8099471529324849 trainer/global_step‚7599 1loss/SRqqt2/assignment_loss‚1.060250278864358 2loss/FBt1/assignment_loss‚0.006947157216909768 /loss/FBt2/assignment_loss‚0.375353642170214 .loss/FRt1/detection_loss‚1.681049498157543 1loss/SRqqt3/detection_loss‚1.8940198585098864 /loss/FBt4/detection_loss‚2.5955122879573276 /loss/FRt2/assignment_loss‚3.032785110613879 1loss/FBt4/assignment_loss‚0.17321491011950588 /loss/FBt1/detection_loss‚0.5662921666242611 .loss/FBt3/detection_loss‚2.317926490306854 1loss/FBt3/assignment_loss‚0.11173509356449358*‘ù]Æ × /loss/FBt4/detection_loss‚2.5955122879573276 /loss/FRt2/assignment_loss‚3.032785110613879 1loss/FBt4/assignment_loss‚0.17321491011950588 /loss/FBt1/detection_loss‚0.5662921666242611 .loss/FBt3/detection_loss‚2.317926490306854 1loss/FBt3/assignment_loss‚0.11173509356449358 .loss/FRt3/detection_loss‚2.516434941742871 /loss/FRt4/detection_loss‚2.8746807306043562 1loss/SRqqt4/detection_loss‚2.1762348930040996 % _timestamp‚1.7884356831641965e+09 .loss/FRt2/detection_loss‚2.286036077667685 &loss/total_loss‚41.779014314551475 epoch‚6 /loss/FRt4/assignment_loss‚3.869626875846617 2loss/SRqqt1/assignment_loss‚0.9833274630408239 1loss/SRqqt3/assignment_loss‚1.633723319891621 2loss/SRqqt4/assignment_loss‚1.5622063751022022 /loss/FRt3/assignment_loss‚2.902262400936436 0loss/SRqqt1/detection_loss‚0.986666855272613 0loss/SRqqt2/detection_loss‚2.133634048539239 /loss/FBt2/detection_loss‚2.2291466858651905 _runtime‚7661.519169093 _step‚625 0loss/FRt1/assignment_loss‚1.8099471529324849 trainer/global_step‚7599 1loss/SRqqt2/assignment_loss‚1.060250278864358 2loss/FBt1/assignment_loss‚0.006947157216909768 /loss/FBt2/assignment_loss‚0.375353642170214 .loss/FRt1/detection_loss‚1.681049498157543 1loss/SRqqt3/detection_loss‚1.8940198585098864ñ-Zªì“Ç ‚(:ˆ æ¹åÔùÍ•µ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚268.668' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê'+íxÈ ‚(:í æ¹åÔ÷•É· network.recv‚ 2.591471e+06# disk.dm-0.out‚62428.86279296875# proc.memory.rssMB‚ 2950.19140625 cpu‚25.27977366252046 network.sent‚ 1.884048e+07 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.986328125" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 287.6484375 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 287.6484375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2080.out‚0$ memory_percent‚99.54168796539307* disk./.usagePercent‚19.059518741062263 disk.drbd2265.out‚0( disk.nvme0c0n1.out‚62428.86279296875 disk.sda3.out‚ 287.6484375* proc.memory.percent‚18.006539344787598% disk./.usageGB‚284.00060272216797& disk.nvme0n1.out‚62428.86279296875 disk.sda3.in‚0.203125 disk.sda1.out‚0' proc.memory.availableMB‚ 75.08984375Ïsn‘É ‚(:† õ¹åÔÜšû´ gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚301.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­YÊÍwÊ ‚(:ì õ¹åÔÕÅå· disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0$ memory_percent‚99.54380989074707" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 288.2734375 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 288.2734375 disk.rbd1.in‚0 disk.dm-0.in‚ 466.90234375& disk.nvme0n1.out‚62504.83056640625) proc.memory.percent‚18.00715923309326 network.sent‚ 1.8949885e+07$ disk./.usageGB‚284.0282440185547# disk.dm-0.out‚62504.83056640625 cpu‚25.279357306441376 proc.cpu.threads‚17 disk.dm-1.in‚0.203125* disk./.usagePercent‚19.061373771655372 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.rbd1.out‚ 3.986328125 disk.drbd2080.in‚0 disk.sda.out‚ 288.2734375 disk.drbd2266.in‚0 network.recv‚ 2.598157e+06( disk.nvme0c0n1.out‚62504.83056640625 disk.drbd2265.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 74.7421875# proc.memory.rssMB‚ 2950.29296875 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0ñ GI“Ë ‚(:ˆ „ºåԪʙµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚289.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø+¨{Ì ‚(:𠄺åÔöŒ· disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 288.58984375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚62591.29931640625 disk.sda3.out‚ 288.58984375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.8955163e+07 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚62591.29931640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 466.90234375* proc.memory.percent‚18.008732795715332& disk.nvme0n1.out‚62591.29931640625# proc.memory.rssMB‚ 2950.55078125 cpu‚25.278916346513615 proc.cpu.threads‚17) disk./.usagePercent‚19.06282405433684 disk.sda.in‚0.203125 disk.sda.out‚ 288.58984375% disk./.usageGB‚284.04985427856445 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.rbd0.out‚0 disk.rbd1.out‚ 3.986328125 disk.dm-1.in‚0.203125# memory_percent‚99.5438814163208 network.recv‚ 2.601542e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 74.73046875" disk.nvme0c0n1.in‚ 466.90234375÷¸Àð2Í - _wandbruntime‚7708  _runtime‚7708…ñ"ÆÎ À _runtime‚7708.415905131 _step‚626 & lr-AdamW/pg1‚7.317589284265173e-05 & lr-AdamW/pg2‚7.317589284265173e-05 trainer/global_step‚7649 $ _timestamp‚1.788435730060839e+09u‘ý—ËÏ Å & lr-AdamW/pg1‚7.317589284265173e-05 & lr-AdamW/pg2‚7.317589284265173e-05 trainer/global_step‚7649 $ _timestamp‚1.788435730060839e+09 _runtime‚7708.415905131 _step‚626ò32Ð - _wandbruntime‚7708  _runtime‚7708ÕH (Ñ ”P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚888456258421248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164763100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16350973905408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60105656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056185107218944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1038361854215207 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16320217017003 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1056188239004160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142198612 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1022038508301312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302006767 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚888456286555648 $ _timestamp‚1.788435730061682e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1039837238707200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82092437 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚911139703241728 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056185107218944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1039837265098752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16347868511744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1039837238707200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16347868511744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22683416686080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22683365865984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164763100 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16350973905408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1038358725318315 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚ 7708.41697994 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137220555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82092440 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1022038533729280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137220561 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137220555 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164786206 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142198093 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1039837265098752 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056185107218944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1038361854215207 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16320217017003 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1056188239004160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142198612 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1022038508301312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302006767 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚888456286555648 $ _timestamp‚1.788435730061682e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1039837238707200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82092437 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚911139703241728 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056185107218944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1039837265098752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16347868511744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1039837238707200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16347868511744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22683416686080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22683365865984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164763100 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16350973905408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1038358725318315 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _runtime‚ 7708.41697994 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137220555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82092440 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1022038533729280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137220561 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137220555 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164786206 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142198093 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1039837265098752 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302045167 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚911255052476416 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1039968759186944 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1038493360009731 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302045167 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚888568669285376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1056321845419520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60113280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164784190 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16349980825600 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 % _timestamp‚1.7884357309601789e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16349980825600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22686304249856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _runtime‚7709.315678204 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142215970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82102690 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚911254973535232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137237839 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60113768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1056318713620992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784K»s@+'Õ ¥N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1039968732795392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22686355056640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164807322 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137237845 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1039968759186944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16353086232576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302045167 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚911255052476416 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1039968759186944 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1038493360009731 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302045167 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚888568669285376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1056321845419520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60113280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164784190 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16349980825600 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 % _timestamp‚1.7884357309601789e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16349980825600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22686304249856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _runtime‚7709.315678204 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142215970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82102690 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚911254973535232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137237839 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60113768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1056318713620992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16322325810103 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137237839 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878ô0ua‘2Ö - _wandbruntime‚7709  _runtime‚7709NÝ ’\× Ö 2loss/SRqqt1/assignment_loss‚0.9853360365342462 2loss/SRqqt3/assignment_loss‚1.4659272741812925 0loss/FBt4/assignment_loss‚0.3769389600493014 .loss/FRt2/detection_loss‚2.281855943955904 /loss/FRt1/detection_loss‚1.5047921535827347 epoch‚6 _runtime‚7709.316935427 2loss/SRqqt2/assignment_loss‚1.4820217478617623 1loss/SRqqt4/detection_loss‚2.2406143963336946 .loss/FBt3/detection_loss‚2.684963583946228 2loss/FBt3/assignment_loss‚0.019942665012257003 /loss/SRqqt2/detection_loss‚2.02095283527632 /loss/FBt2/detection_loss‚2.2205206339175882 1loss/SRqqt3/detection_loss‚1.5785710990428925 0loss/FRt4/assignment_loss‚3.7993952311002293 .loss/FRt3/detection_loss‚2.280809694102832 /loss/FRt4/detection_loss‚2.8501415237402306 1loss/SRqqt1/detection_loss‚1.0946315523857872 0loss/FRt1/assignment_loss‚1.5734636965444533 0loss/FRt2/assignment_loss‚3.5588049418047856 2loss/SRqqt4/assignment_loss‚1.8720501132309437 0loss/FBt2/assignment_loss‚0.3247081166706406 /loss/FBt4/detection_loss‚2.7058699826399484 %loss/total_loss‚42.12963372456541 .loss/FBt1/detection_loss‚0.639626600198886 /loss/FRt3/assignment_loss‚2.555943252075286 2loss/FBt1/assignment_loss‚0.011751690377165523 _step‚629 trainer/global_step‚7649 % _timestamp‚1.7884357309619331e+09F¿{ÏaØ Û .loss/FBt1/detection_loss‚0.639626600198886 /loss/FRt3/assignment_loss‚2.555943252075286 2loss/FBt1/assignment_loss‚0.011751690377165523 _step‚629 trainer/global_step‚7649 % _timestamp‚1.7884357309619331e+09 2loss/SRqqt1/assignment_loss‚0.9853360365342462 2loss/SRqqt3/assignment_loss‚1.4659272741812925 0loss/FBt4/assignment_loss‚0.3769389600493014 .loss/FRt2/detection_loss‚2.281855943955904 /loss/FRt1/detection_loss‚1.5047921535827347 epoch‚6 _runtime‚7709.316935427 2loss/SRqqt2/assignment_loss‚1.4820217478617623 1loss/SRqqt4/detection_loss‚2.2406143963336946 .loss/FBt3/detection_loss‚2.684963583946228 2loss/FBt3/assignment_loss‚0.019942665012257003 /loss/SRqqt2/detection_loss‚2.02095283527632 /loss/FBt2/detection_loss‚2.2205206339175882 1loss/SRqqt3/detection_loss‚1.5785710990428925 0loss/FRt4/assignment_loss‚3.7993952311002293 .loss/FRt3/detection_loss‚2.280809694102832 /loss/FRt4/detection_loss‚2.8501415237402306 1loss/SRqqt1/detection_loss‚1.0946315523857872 0loss/FRt1/assignment_loss‚1.5734636965444533 0loss/FRt2/assignment_loss‚3.5588049418047856 2loss/SRqqt4/assignment_loss‚1.8720501132309437 0loss/FBt2/assignment_loss‚0.3247081166706406 /loss/FBt4/detection_loss‚2.7058699826399484 %loss/total_loss‚42.12963372456541õt“Ù ‚(:ˆ “ºåÔŽ–õ gpu.0.gpu‚82.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚281.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vº~xÚ ‚(:í “ºåÔ‘Áß· disk.rbd1.out‚ 3.986328125 disk.sda3.in‚0.203125 disk.sda3.out‚ 289.26953125 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375" disk.dm-0.out‚62672.8486328125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚62672.8486328125& proc.memory.availableMB‚ 74.6484375# proc.memory.rssMB‚ 2950.59765625 network.sent‚ 1.8961221e+07% disk./.usageGB‚284.07375717163086 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0) disk./.usagePercent‚19.06442819754639% disk.nvme0n1.out‚62672.8486328125 disk.drbd2266.out‚0* proc.memory.percent‚18.009018898010254 proc.cpu.threads‚17 disk.dm-1.out‚ 289.26953125 cpu‚25.278468326266406 network.recv‚ 2.606116e+06 disk.nvme0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 289.26953125$ memory_percent‚99.54438209533691 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0¼eê„žÛ ‚(:“ ¢ºåÔÚñ¥µ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚284.839' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.967800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ<=wÜ ‚(:ì ¢ºåÔ½çÁ¸ disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0.203125 cpu‚25.278020651072225) disk./.usagePercent‚19.06568417050937# disk.dm-0.out‚62747.68798828125 disk.nvme0n1.in‚ 466.90234375 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2950.6953125* proc.memory.percent‚18.009614944458008 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.dm-1.out‚ 289.66015625 network.sent‚ 1.9069866e+07 disk.sda1.in‚0 disk.sda.out‚ 289.66015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.986328125$ memory_percent‚99.54652786254883 network.recv‚ 2.612217e+06 disk.dm-0.in‚ 466.90234375 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 466.90234375# disk./.usageGB‚284.092472076416 disk.sda3.out‚ 289.66015625 disk.drbd2080.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚62747.68798828125% proc.memory.availableMB‚ 74.296875 disk.sda3.in‚0.203125 disk.rbd0.in‚0& disk.nvme0n1.out‚62747.68798828125.”Û“Ý ‚(:ˆ ±ºåÔ¨î͵ gpu.0.gpu‚51.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚4 KxÞ ‚(:í ±ºåÔ³ê¶& proc.memory.availableMB‚ 74.4609375) proc.memory.percent‚18.01018714904785 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚62817.25634765625 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 3.986328125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 cpu‚25.277591677259384 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 network.sent‚ 1.9076271e+07$ disk./.usageGB‚284.1071968078613 disk.drbd2266.out‚0 disk.sda3.out‚ 290.26171875 disk.sda1.in‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0# memory_percent‚99.5455265045166" proc.memory.rssMB‚ 2950.7890625* disk./.usagePercent‚19.066672359591543 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 290.26171875 network.recv‚ 2.617189e+06 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚62817.25634765625 disk.sda3.in‚0.203125 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚62817.25634765625 disk.sda2.in‚0 disk.dm-1.out‚ 290.26171875BÚ%V“ß ‚(:ˆ ÀºåÔóƵ gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚132.881' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø¯ërà ‚(:ç ÀºåÔß›ò¶ disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125$ memory_percent‚99.54686164855957 disk.rbd0.in‚0 disk.sda3.out‚ 290.71875 cpu‚25.277120382183107 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚62862.74072265625 disk.sda.out‚ 290.71875 proc.cpu.threads‚17 disk.sda.in‚0.203125& disk.nvme0n1.out‚62862.74072265625 disk.dm-1.out‚ 290.71875 network.recv‚ 2.620771e+06* disk./.usagePercent‚19.065609416309375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚62862.74072265625& proc.memory.availableMB‚ 74.2421875 network.sent‚ 1.9081602e+07 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375# proc.memory.rssMB‚ 2950.99609375) proc.memory.percent‚18.01145076751709% disk./.usageGB‚284.09135818481445 disk.sda2.in‚0­ÔnÓ2á - _wandbruntime‚7755  _runtime‚7755RÁ8cÇâ Á & lr-AdamW/pg1‚7.176343001306236e-05 & lr-AdamW/pg2‚7.176343001306236e-05 trainer/global_step‚7699 % _timestamp‚1.7884357775704675e+09 _runtime‚7755.925560182 _step‚6302à¿ÏÌã Æ & lr-AdamW/pg1‚7.176343001306236e-05 & lr-AdamW/pg2‚7.176343001306236e-05 trainer/global_step‚7699 % _timestamp‚1.7884357775704675e+09 _runtime‚7755.925560182 _step‚630ö1MÅ 2ä - _wandbruntime‚7755  _runtime‚7755+ÿ¥`Úå –P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138084755 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22830813050880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚631 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1062865441735168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1028508383790080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1062868573520384 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1046411943116800 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1062868573520384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1062865441735168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚894076801627648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16456604012032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1044937157522215 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 trainer/global_step‚7699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60488848 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1028508358362112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.¾%õ‹Ballocated‚916907693633024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16453498618368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60488330 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1044934028625323 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138084755 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143093417 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138084761 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303905707 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82605090 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚7755.926580478 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165844058 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165820952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884357775712712e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303928819 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1046411969508352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16456604012032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16453498618368 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22830863870976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165844058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82605087 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16425670263211 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165820952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1046411969508352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚894076829762048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16428773732135 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1046411943116800ÃgC !(æ ›P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚916907693633024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16453498618368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60488330 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1044934028625323 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138084755 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143093417 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138084761 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303905707 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82605090 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚7755.926580478 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165844058 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165820952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884357775712712e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303928819 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1046411969508352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16456604012032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16453498618368 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22830863870976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165844058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82605087 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16425670263211 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165820952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1046411969508352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚894076829762048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16428773732135 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1046411943116800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138084755 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22830813050880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚631 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1062865441735168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1028508383790080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1062868573520384 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1046411943116800 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1062868573520384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1062865441735168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚894076801627648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16456604012032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1044937157522215 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 trainer/global_step‚7699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60488848 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1028508358362112÷5&üp2ç - _wandbruntime‚7756  _runtime‚7756mw÷&'è  N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1028637780791296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚917022977793024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16430882525443 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303967219 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚894189240626176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143111951 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60496117 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16455610932224 % _timestamp‚1.7884357784648883e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22833816108032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1046543463596544 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165842042 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16427779056311 _runtime‚7756.820826633 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16455610932224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60496608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1063002179935744 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303944081 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1046543463596544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138102045 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22833765301248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16458716339200 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143111457 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚7699 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138102039 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165865174 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303944081 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1045065534419639 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16458716339200 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1046543437204992 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82615340 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1062999048137216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165842042 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1045068663316739î…÷"é ¥N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1063002179935744 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303944081 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1046543463596544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138102045 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22833765301248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16458716339200 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143111457 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚7699 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138102039 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165865174 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303944081 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1045065534419639 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16458716339200 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1046543437204992 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82615340 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1062999048137216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165842042 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1045068663316739 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1028637780791296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚917022977793024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16430882525443 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303967219 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚894189240626176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143111951 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60496117 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16455610932224 % _timestamp‚1.7884357784648883e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22833816108032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1046543463596544 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165842042 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16427779056311 _runtime‚7756.820826633 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16455610932224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60496608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968ø·Ò°2ê - _wandbruntime‚7756  _runtime‚7756lœìYë Ó 0loss/FRt1/assignment_loss‚1.2955222670848554 /loss/FRt3/assignment_loss‚3.209653158982595 2loss/SRqqt3/assignment_loss‚1.2084398035025388 2loss/FBt1/assignment_loss‚0.029469442708683968 /loss/FRt4/detection_loss‚3.0900485600743974 0loss/SRqqt2/detection_loss‚2.500607337682478 2loss/SRqqt4/assignment_loss‚1.6630366295576096 0loss/FBt3/assignment_loss‚0.6716629947477486 /loss/FRt1/detection_loss‚1.5363739389066513 1loss/SRqqt1/detection_loss‚0.8107141491112321 .loss/FBt2/detection_loss‚2.289228677749634 /loss/FRt4/assignment_loss‚4.196265361138752 2loss/SRqqt1/assignment_loss‚1.2651063047111208 1loss/FBt4/assignment_loss‚0.11889507972324888 /loss/FRt2/detection_loss‚2.4689508325014358 0loss/SRqqt3/detection_loss‚1.715699827461912 /loss/FBt3/detection_loss‚3.0097800716757774 %loss/total_loss‚45.04723706205759 _step‚633 0loss/FRt2/assignment_loss‚3.7899104356765747 /loss/FRt3/detection_loss‚2.3978015578455394 0loss/SRqqt4/detection_loss‚2.340374862536406 /loss/FBt1/detection_loss‚0.7087949120088831 .loss/FBt4/detection_loss‚2.226524488131205 epoch‚6 $ _timestamp‚1.788435778467377e+09 0loss/SRqqt2/assignment_loss‚2.30672290422503 1loss/FBt2/assignment_loss‚0.19765346431328604 trainer/global_step‚7699 _runtime‚7756.822369405Á_Ä9^ì Ø /loss/FRt4/assignment_loss‚4.196265361138752 2loss/SRqqt1/assignment_loss‚1.2651063047111208 1loss/FBt4/assignment_loss‚0.11889507972324888 /loss/FRt2/detection_loss‚2.4689508325014358 0loss/SRqqt3/detection_loss‚1.715699827461912 /loss/FBt3/detection_loss‚3.0097800716757774 %loss/total_loss‚45.04723706205759 _step‚633 0loss/FRt2/assignment_loss‚3.7899104356765747 /loss/FRt3/detection_loss‚2.3978015578455394 0loss/SRqqt4/detection_loss‚2.340374862536406 /loss/FBt1/detection_loss‚0.7087949120088831 .loss/FBt4/detection_loss‚2.226524488131205 epoch‚6 $ _timestamp‚1.788435778467377e+09 0loss/SRqqt2/assignment_loss‚2.30672290422503 1loss/FBt2/assignment_loss‚0.19765346431328604 trainer/global_step‚7699 _runtime‚7756.822369405 0loss/FRt1/assignment_loss‚1.2955222670848554 /loss/FRt3/assignment_loss‚3.209653158982595 2loss/SRqqt3/assignment_loss‚1.2084398035025388 2loss/FBt1/assignment_loss‚0.029469442708683968 /loss/FRt4/detection_loss‚3.0900485600743974 0loss/SRqqt2/detection_loss‚2.500607337682478 2loss/SRqqt4/assignment_loss‚1.6630366295576096 0loss/FBt3/assignment_loss‚0.6716629947477486 /loss/FRt1/detection_loss‚1.5363739389066513 1loss/SRqqt1/detection_loss‚0.8107141491112321 .loss/FBt2/detection_loss‚2.289228677749634ùöTÜÆží ‚(:“ ϺåÔ¢×›µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚284.639' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.927800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ëCi{î ‚(:ð ϺåÔäµ· disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2951.04296875 disk.sda1.out‚0( disk.nvme0c0n1.out‚62945.62158203125 disk.drbd1716.out‚0$ disk./.usageGB‚284.1295585632324 disk.rbd1.out‚ 3.986328125 disk.sda3.in‚0.203125) proc.memory.percent‚18.01173686981201 cpu‚25.276679871787284 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda.out‚ 291.30078125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 291.30078125 network.sent‚ 1.9087627e+07 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 74.41015625* disk./.usagePercent‚19.068173075757258 disk.rbd0.in‚0& disk.nvme0n1.out‚62945.62158203125$ memory_percent‚99.54583644866943 network.recv‚ 2.625064e+06# disk.dm-0.out‚62945.62158203125 disk.sda3.out‚ 291.30078125 disk.sda1.in‚0 disk.nvme0n1.in‚ 466.90234375" disk.nvme0c0n1.in‚ 466.902343757ÐnM“ï ‚(:ˆ ÞºåÔ«…ßµ gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚211.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.2758 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àu­;rð ‚(:ç ÞºåÔÅÈì¶ network.recv‚ 2.631173e+06 disk.dm-1.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚63018.62158203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 disk.sda3.in‚0.203125# proc.memory.rssMB‚ 2951.18359375 proc.cpu.threads‚17 disk.dm-1.out‚ 291.734375 disk.drbd1716.out‚0 disk.drbd2080.out‚0$ memory_percent‚99.54671859741211 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.sda3.out‚ 291.734375 disk.sda.out‚ 291.734375 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375* proc.memory.percent‚18.012595176696777$ disk./.usageGB‚284.1535873413086( disk.nvme0c0n1.out‚63018.62158203125 disk.sda1.out‚0 disk.drbd1716.in‚0% proc.memory.availableMB‚ 74.265625 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚63018.62158203125 network.sent‚ 1.919626e+07* disk./.usagePercent‚19.069785667215438 cpu‚25.27624833108011­wž“ñ ‚(:ˆ íºåÔîõ´ gpu.0.gpu‚69.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.9956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ø!pò ‚(:å íºåÔ±·›¸ disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 74.16015625# proc.memory.rssMB‚ 2951.28515625 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚63097.35107421875 proc.cpu.threads‚17 network.sent‚ 1.9202721e+07 network.recv‚ 2.635936e+06) disk./.usagePercent‚19.06876675722916 disk.rbd1.out‚ 3.986328125$ disk./.usageGB‚284.1384048461914 disk.dm-0.in‚ 466.90234375 disk.sda1.out‚0 disk.sda.out‚ 292.59375 disk.sda2.in‚0# disk.dm-0.out‚63097.35107421875 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 292.59375 disk.drbd2266.in‚0 disk.rbd1.in‚0) proc.memory.percent‚18.01321506500244 disk.sda1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚63097.35107421875 disk.drbd2266.out‚0$ memory_percent‚99.54736232757568 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.sda3.out‚ 292.59375 disk.drbd1716.in‚0" disk.nvme0c0n1.in‚ 466.90234375 cpu‚25.27583751539811 disk.sda3.in‚0.203125JDÂÄ2ó - _wandbruntime‚7803  _runtime‚7803Fü2Äô ¾ trainer/global_step‚7749 $ _timestamp‚1.788435825310011e+09 _runtime‚7803.665049626 _step‚634 % lr-AdamW/pg1‚7.03573278773631e-05 % lr-AdamW/pg2‚7.03573278773631e-05%e†‡Éõ à _step‚634 % lr-AdamW/pg1‚7.03573278773631e-05 % lr-AdamW/pg2‚7.03573278773631e-05 trainer/global_step‚7749 $ _timestamp‚1.788435825310011e+09 _runtime‚7803.665049626úÖÝi2ö - _wandbruntime‚7803  _runtime‚78036¼´(÷ •P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16534226978343 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16531123509419 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1052986647526400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1034978208422912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1069548908036608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1069548908036608 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1069545776251392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚922675516791808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1052986673917952 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1069545776251392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1052986647526400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143991019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305827759 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22978088808960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60873282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16562234118656 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16562234118656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚899697372968448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138948955 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166878804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83117740 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1034978233850880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16559128724992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166901910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1051512460829223 _runtime‚7803.666037961 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚899697344834048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚635 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138948955 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22978143823360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138948961 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166901910 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166878804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83117740 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1034978233850880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16559128724992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166901910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1051512460829223 _runtime‚7803.666037961 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚899697344834048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚635 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138948955 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22978143823360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138948961 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166901910 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16534226978343 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16531123509419 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1052986647526400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1034978208422912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1069548908036608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1069548908036608 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1069545776251392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚922675516791808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1052986673917952 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1069545776251392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1052986647526400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143991019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305827759 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22978088808960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60873282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16562234118656 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16562234118656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚899697372968448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138948955 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60880841 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138966245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚899809783832576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166899894 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144009380 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138966245 _runtime‚7804.552847776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138966239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16533232302519 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚922790782471168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚899809755698176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83127993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1053118141614592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1051640837726647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305889271 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1069682514451968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22981081774080 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1053118168006144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144008831 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1035107605424128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16536335771651 _step‚636 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1069682514451968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166899894 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138966239 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305889271 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1035107630852096—;#+'û ¥N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1069682514451968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22981081774080 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1053118168006144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144008831 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1035107605424128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16536335771651 _step‚636 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1069682514451968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166899894 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138966239 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305889271 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1035107630852096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83127990 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22981026772992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884358261970026e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60880841 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138966245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚899809783832576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166899894 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144009380 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138966245 _runtime‚7804.552847776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138966239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16533232302519 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚922790782471168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚899809755698176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83127993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1053118141614592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1051640837726647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305889271üŸÛo2ü - _wandbruntime‚7804  _runtime‚7804È?MR_ý Ù 0loss/FRt2/assignment_loss‚3.5126782011341406 /loss/FRt3/detection_loss‚2.2907916443779115 0loss/SRqqt3/detection_loss‚1.572487851488404 0loss/SRqqt4/detection_loss‚2.483706001440684 /loss/FBt3/detection_loss‚2.6805076636373997 _step‚637 2loss/SRqqt1/assignment_loss‚0.9770835428319585 2loss/SRqqt4/assignment_loss‚1.3735013430317242 2loss/SRqqt3/assignment_loss‚1.3594517026795074 1loss/FBt2/assignment_loss‚0.15483017209252076 .loss/FRt1/detection_loss‚1.628718237938552 /loss/FBt2/detection_loss‚2.5052066019603183 epoch‚6 trainer/global_step‚7749 % _timestamp‚1.7884358261990614e+09 /loss/FRt1/assignment_loss‚1.564285088202049 0loss/FRt3/assignment_loss‚2.7610069411866207 2loss/FBt1/assignment_loss‚0.008222322086072535 /loss/FRt4/detection_loss‚2.9494994628813958 %loss/total_loss‚42.19734864670853 0loss/FRt4/assignment_loss‚3.7981859137935023 2loss/SRqqt2/assignment_loss‚1.7037672512233257 1loss/SRqqt2/detection_loss‚2.3366335295140743 1loss/FBt3/assignment_loss‚0.10297150822589174 /loss/FBt1/detection_loss‚0.5052692940724748 2loss/FBt4/assignment_loss‚0.006338777868742389 .loss/FRt2/detection_loss‚2.386727558599936 1loss/SRqqt1/detection_loss‚1.0500138381925912 .loss/FBt4/detection_loss‚2.485464198248727 _runtime‚7804.554035467´+’Ûdþ Þ 2loss/FBt4/assignment_loss‚0.006338777868742389 .loss/FRt2/detection_loss‚2.386727558599936 1loss/SRqqt1/detection_loss‚1.0500138381925912 .loss/FBt4/detection_loss‚2.485464198248727 _runtime‚7804.554035467 0loss/FRt2/assignment_loss‚3.5126782011341406 /loss/FRt3/detection_loss‚2.2907916443779115 0loss/SRqqt3/detection_loss‚1.572487851488404 0loss/SRqqt4/detection_loss‚2.483706001440684 /loss/FBt3/detection_loss‚2.6805076636373997 _step‚637 2loss/SRqqt1/assignment_loss‚0.9770835428319585 2loss/SRqqt4/assignment_loss‚1.3735013430317242 2loss/SRqqt3/assignment_loss‚1.3594517026795074 1loss/FBt2/assignment_loss‚0.15483017209252076 .loss/FRt1/detection_loss‚1.628718237938552 /loss/FBt2/detection_loss‚2.5052066019603183 epoch‚6 trainer/global_step‚7749 % _timestamp‚1.7884358261990614e+09 /loss/FRt1/assignment_loss‚1.564285088202049 0loss/FRt3/assignment_loss‚2.7610069411866207 2loss/FBt1/assignment_loss‚0.008222322086072535 /loss/FRt4/detection_loss‚2.9494994628813958 %loss/total_loss‚42.19734864670853 0loss/FRt4/assignment_loss‚3.7981859137935023 2loss/SRqqt2/assignment_loss‚1.7037672512233257 1loss/SRqqt2/detection_loss‚2.3366335295140743 1loss/FBt3/assignment_loss‚0.10297150822589174 /loss/FBt1/detection_loss‚0.5052692940724748ý;wiÏÿ ‚(:’ üºåÔ‹ÿ´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚283.153' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.63060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X“o€!‚(:ä üºåÔÉ䤷 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚63172.70263671875 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 293.15625' proc.memory.availableMB‚ 73.98828125 disk.sda1.out‚0 disk.drbd2080.out‚0 network.sent‚ 1.9207999e+07 network.recv‚ 2.6394e+06& disk.nvme0n1.out‚63172.70263671875# disk.dm-0.out‚63172.70263671875 disk.sda3.in‚0.203125 disk.sda3.out‚ 293.15625 disk.sda1.in‚0) disk./.usagePercent‚19.07027899373392 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2951.28515625) proc.memory.percent‚18.01321506500244 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.986328125 disk.rbd0.in‚0 cpu‚25.27537979116166 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-1.out‚ 293.15625 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0% disk./.usageGB‚284.16093826293945 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 466.90234375$ memory_percent‚99.54841136932373®Ìé>ž!‚(:“ ‹»åÔ¸Ïæµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.217600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œIØ3w‚!‚(:ì ‹»åÔéɯ·$ disk./.usageGB‚284.1923484802246 disk.dm-0.in‚ 466.90234375 disk.rbd0.in‚0# memory_percent‚99.5492696762085) proc.memory.percent‚18.01321506500244 disk.sda3.out‚ 293.8671875 disk.drbd2080.out‚0 disk.sda.in‚0.203125( disk.nvme0c0n1.out‚63256.30126953125 disk.dm-1.in‚0.203125# disk.dm-0.out‚63256.30126953125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚63256.30126953125 network.sent‚ 1.9316835e+07 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 cpu‚25.274948834835303 proc.cpu.threads‚17 network.recv‚ 2.646046e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda.out‚ 293.8671875* disk./.usagePercent‚19.072386959770853 disk.dm-1.out‚ 293.8671875 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 73.84765625# proc.memory.rssMB‚ 2951.28515625+,8Ç“ƒ!‚(:ˆ š»åÔ•Øš¶ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚290.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.1812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R«yl„!‚(:á š»åÔɇç¶ disk.rbd1.in‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚63327.77392578125 disk.sda3.in‚0.203125 disk.rbd0.out‚0* proc.memory.percent‚18.013501167297363 cpu‚25.274523904728703 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚63327.77392578125 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2951.33203125 disk.sda3.out‚294.375 disk.sda1.in‚0 disk.drbd2080.out‚0$ memory_percent‚99.54955577850342 network.recv‚ 2.650039e+06( disk.nvme0c0n1.out‚63327.77392578125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.sent‚ 1.9322703e+07" disk.nvme0c0n1.in‚ 466.90234375 proc.cpu.threads‚17* disk./.usagePercent‚19.074207941361905 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚294.375 disk.drbd2266.out‚0 disk.sda.out‚294.375 disk.rbd0.in‚0' proc.memory.availableMB‚ 73.80078125# disk./.usageGB‚284.219482421875 disk.rbd1.out‚ 3.986328125 disk.sda1.out‚0 disk.sda.in‚0.203125Ÿ0h×2…!- _wandbruntime‚7850  _runtime‚7850·xdž!Á & lr-AdamW/pg1‚6.895787946513156e-05 & lr-AdamW/pg2‚6.895787946513156e-05 trainer/global_step‚7799 % _timestamp‚1.7884358724337575e+09 _runtime‚7850.788782239 _step‚638‡@V=̇!Æ & lr-AdamW/pg1‚6.895787946513156e-05 & lr-AdamW/pg2‚6.895787946513156e-05 trainer/global_step‚7799 % _timestamp‚1.7884358724337575e+09 _runtime‚7850.788782239 _step‚638þÄ$©2ˆ!- _wandbruntime‚7850  _runtime‚7850Û;jfD‰!–P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1059561378327552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16664758831616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1058084635239339 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884358724345088e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1041448058483712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139813155 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚905317888040448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1059561351936000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61255143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23124844948480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚7850.789767889 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1076229242552832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144886066 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1041448083911680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307749811 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16636576755627 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167959762 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167936656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139813155 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83630387 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928442814040576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚639 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16664758831616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚7799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139813161 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚928442732988928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167936656 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1076226110767616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307772923 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚905317916174848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144886066 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1041448083911680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307749811 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16636576755627 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167959762 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167936656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139813155 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83630387 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928442814040576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚639 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16664758831616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚7799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139813161 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚928442732988928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167936656 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1076226110767616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307772923 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚905317916174848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139813155 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚905317888040448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1059561351936000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61255143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23124844948480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚7850.789767889 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1076229242552832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307788185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144903872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚905430298904576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167980878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16669976552448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23127826086400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23127773182464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167957746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928558153125376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058216141033655 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1041577480912896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83640640 trainer/global_step‚7799 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139830445 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16669976552448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16666871145472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16666871145472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚905430327038976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307811323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61263229 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058219269930755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1059692872415744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307788185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16641789017859 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144903356 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16638685548727 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6h Z/Œ!¥N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307811323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61263229 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058219269930755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1059692872415744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307788185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16641789017859 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144903356 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16638685548727 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1076362848968192 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307788185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144903872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚905430298904576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167980878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16669976552448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23127826086400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23127773182464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167957746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928558153125376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058216141033655 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1041577480912896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83640640 trainer/global_step‚7799 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139830445 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16669976552448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16666871145472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _step‚640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16666871145472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚905430327038976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936€%!v.2Ž!- _wandbruntime‚7851  _runtime‚7851«¿ÜX!Ò 0loss/FRt3/assignment_loss‚2.7706131781850543 0loss/SRqqt2/detection_loss‚2.616908708810806 .loss/FBt1/detection_loss‚0.475950818327142 % _timestamp‚1.7884358733364506e+09 /loss/FRt2/assignment_loss‚3.196576379440926 1loss/SRqqt3/assignment_loss‚1.588440124933817 0loss/SRqqt4/detection_loss‚1.869420635394561 %loss/total_loss‚41.16742614461815 trainer/global_step‚7799 2loss/SRqqt1/assignment_loss‚1.0070883109960909 1loss/FBt3/assignment_loss‚0.29052885310816884 0loss/FBt4/assignment_loss‚0.2578135963477608 .loss/FRt3/detection_loss‚2.419307770047869 .loss/FRt4/detection_loss‚2.742684935369799 1loss/SRqqt2/assignment_loss‚1.274822643995285 /loss/FBt4/detection_loss‚1.9071074608299468 /loss/FBt2/detection_loss‚2.6551749885082243 1loss/SRqqt1/detection_loss‚0.8746522372731795 _runtime‚7851.691416481 _step‚641 /loss/FBt3/detection_loss‚2.6020299111093794 1loss/SRqqt4/assignment_loss‚1.278307334591563 2loss/FBt1/assignment_loss‚0.010552556717986397 1loss/FBt2/assignment_loss‚0.05511376797221601 0loss/FRt4/assignment_loss‚3.9228483246218775 0loss/SRqqt3/detection_loss‚1.933099622477074 epoch‚6 .loss/FRt1/detection_loss‚1.519369079019217 /loss/FRt2/detection_loss‚2.7409492666656905 0loss/FRt1/assignment_loss‚1.1580656398745144³>¡]!× _runtime‚7851.691416481 _step‚641 /loss/FBt3/detection_loss‚2.6020299111093794 1loss/SRqqt4/assignment_loss‚1.278307334591563 2loss/FBt1/assignment_loss‚0.010552556717986397 1loss/FBt2/assignment_loss‚0.05511376797221601 0loss/FRt4/assignment_loss‚3.9228483246218775 0loss/SRqqt3/detection_loss‚1.933099622477074 epoch‚6 .loss/FRt1/detection_loss‚1.519369079019217 /loss/FRt2/detection_loss‚2.7409492666656905 0loss/FRt1/assignment_loss‚1.1580656398745144 0loss/FRt3/assignment_loss‚2.7706131781850543 0loss/SRqqt2/detection_loss‚2.616908708810806 .loss/FBt1/detection_loss‚0.475950818327142 % _timestamp‚1.7884358733364506e+09 /loss/FRt2/assignment_loss‚3.196576379440926 1loss/SRqqt3/assignment_loss‚1.588440124933817 0loss/SRqqt4/detection_loss‚1.869420635394561 %loss/total_loss‚41.16742614461815 trainer/global_step‚7799 2loss/SRqqt1/assignment_loss‚1.0070883109960909 1loss/FBt3/assignment_loss‚0.29052885310816884 0loss/FBt4/assignment_loss‚0.2578135963477608 .loss/FRt3/detection_loss‚2.419307770047869 .loss/FRt4/detection_loss‚2.742684935369799 1loss/SRqqt2/assignment_loss‚1.274822643995285 /loss/FBt4/detection_loss‚1.9071074608299468 /loss/FBt2/detection_loss‚2.6551749885082243 1loss/SRqqt1/detection_loss‚0.8746522372731795k j‘!‚(:’ ©»åÔö½´µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚275.382' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.07640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tUaw’!‚(:ì ©»åÔ;Ÿ·' disk.nvme0c0n1.out‚63426.6279296875 disk.sda3.out‚ 295.05078125 disk.drbd2080.out‚0 disk.drbd2265.out‚0) disk./.usagePercent‚19.07588479071118 disk.sda2.out‚0 network.recv‚ 2.654787e+06 disk.dm-1.out‚ 295.05078125$ memory_percent‚99.55143928527832& proc.memory.availableMB‚ 73.4921875# proc.memory.rssMB‚ 2951.33203125* proc.memory.percent‚18.013501167297363 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.986328125 disk.drbd2080.in‚0 disk.sda.out‚ 295.05078125 disk.drbd2265.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0.203125" disk.dm-0.out‚63426.6279296875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 network.sent‚ 1.9329018e+07" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.27412558343427 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125% disk./.usageGB‚284.24446868896484% disk.nvme0n1.out‚63426.6279296875 disk.sda2.in‚0ÎÈý:‘“!‚(:† ¸»åÔÚ¥‘µ gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^<'ét”!‚(:é ¸»åÔŽÝè¶ disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda.in‚0.203125# memory_percent‚99.5502233505249 disk.dm-1.out‚ 295.4453125 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚63490.0146484375" disk.dm-0.out‚63490.0146484375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 295.4453125% disk.nvme0n1.out‚63490.0146484375' proc.memory.availableMB‚ 73.69140625 cpu‚25.27370291280188 proc.cpu.threads‚17 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2951.3828125 network.recv‚ 2.660673e+06* disk./.usagePercent‚19.077505318402917 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0* proc.memory.percent‚18.013811111450195 disk.drbd1716.in‚0 disk.sda.out‚ 295.4453125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 network.sent‚ 1.9437164e+07% disk./.usageGB‚284.26861572265625 disk.dm-0.in‚ 466.90234375 disk.sda1.out‚0dd‘ž•!‚(:“ Ç»åÔ¯¡ãµ gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚278.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.604200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Ê0"t–!‚(:é Ç»åԆݞ· disk.rbd1.out‚ 4.072265625 disk.sda.out‚ 296.30078125 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 296.30078125 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2951.4296875 proc.cpu.threads‚17# disk.dm-0.out‚63569.92041015625 disk.rbd0.in‚0& disk.nvme0n1.out‚63569.92041015625 disk.dm-0.in‚ 466.90234375 disk.drbd1716.out‚0 disk.drbd2080.in‚0$ memory_percent‚99.55205917358398 cpu‚25.2732446504285 disk.sda2.out‚0 disk.dm-1.out‚ 296.30078125 network.sent‚ 1.944335e+07 network.recv‚ 2.665268e+06 disk.drbd2080.out‚0* proc.memory.percent‚18.014097213745117* disk./.usagePercent‚19.078648136035767" disk./.usageGB‚284.28564453125 disk.rbd0.out‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚63569.92041015625 disk.sda3.in‚0.203125 disk.sda1.in‚0% proc.memory.availableMB‚ 73.390625 disk.sda2.in‚0 disk.drbd2266.in‚0tňn2—!- _wandbruntime‚7897  _runtime‚7897ÿg3Hǘ!Á & lr-AdamW/pg1‚6.756537641931929e-05 & lr-AdamW/pg2‚6.756537641931929e-05 trainer/global_step‚7849 % _timestamp‚1.7884359196138754e+09 _runtime‚7897.968958199 _step‚642 cyiÌ™!Æ & lr-AdamW/pg1‚6.756537641931929e-05 & lr-AdamW/pg2‚6.756537641931929e-05 trainer/global_step‚7849 % _timestamp‚1.7884359196138754e+09 _runtime‚7897.968958199 _step‚642‚ö1Ä2š!- _wandbruntime‚7897  _runtime‚7897J×Q(›!•P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309694975 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145781570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61638530 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16745133470759 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1047917908544512 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1082906445283840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1047917933972480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1082909577069056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7849 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1082906445283840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1066136082737152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1066136056345600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169017614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84143040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚910938459381248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145781019 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚934209718974976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84143037 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309671863 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1064659938546347 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140677361 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚910938431246848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1082909577069056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23271287728128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16773494331904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16773494331904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140677361 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1066136082737152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _step‚643 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140677355 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169017614 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16770388938240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61637982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168994508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚934209802123776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1066136056345600 $ _timestamp‚1.788435919614658e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16770388938240 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23271342742528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚7897.969950391 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1064663067443239 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302V0ñ (œ!šP ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1066136056345600 $ _timestamp‚1.788435919614658e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16770388938240 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23271342742528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚7897.969950391 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1064663067443239 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309694975 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145781570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61638530 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16745133470759 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1047917908544512 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1082906445283840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1047917933972480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1082909577069056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7849 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1082906445283840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1066136082737152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1066136056345600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169017614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84143040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚910938459381248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145781019 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚934209718974976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84143037 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309671863 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1064659938546347 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140677361 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚910938431246848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1082909577069056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23271287728128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16773494331904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16773494331904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140677361 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1066136082737152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _step‚643 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140677355 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169017614 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16770388938240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61637982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168994508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚934209802123776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032ƒ½Žæ2!- _wandbruntime‚7898  _runtime‚7898lFsmž! N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309733375 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309710237 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚911050870245376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16775606659072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169015598 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trš(W(¹"ainer/global_step‚7849 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1066267550433792 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1066267550433792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚934325150072832 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140694645 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145799448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23274224826368 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140694639 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309733375 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16747242264067 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1066267576825344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1066267576825344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145798908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1083043183484416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61646155 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169038730 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16772501252096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1083043183484416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚7898.863304474 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1048047330973696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 % _timestamp‚1.7884359205075042e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1083043183484416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _runtime‚7898.863304474 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1048047330973696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 % _timestamp‚1.7884359205075042e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140694639 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309733375 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16747242264067 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1066267576825344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1066267576825344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145798908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1083043183484416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0„ù sê2 !- _wandbruntime‚7898  _runtime‚7898¢”óL^¡!Ø 0loss/FRt2/assignment_loss‚3.1071214520412944 epoch‚6 1loss/FBt2/assignment_loss‚0.12006868949780466 .loss/FRt2/detection_loss‚2.169061642626157 /loss/FRt4/detection_loss‚2.9435088272605623 1loss/SRqqt2/detection_loss‚2.0220147325442386 $ _timestamp‚1.788435920509428e+09 /loss/FRt3/assignment_loss‚2.800451846349807 2loss/SRqqt1/assignment_loss‚0.7817445444719245 2loss/SRqqt3/assignment_loss‚1.2716626702403684 /loss/FRt1/detection_loss‚1.7168301933014125 1loss/SRqqt1/detection_loss‚0.9832073503972164 trainer/global_step‚7849 _runtime‚7898.864426963 3loss/FBt1/assignment_loss‚0.0023255506431950294 /loss/FRt3/detection_loss‚2.4591737056062337 /loss/FBt1/detection_loss‚0.6514424632973316 0loss/FRt1/assignment_loss‚1.7347702259448037 .loss/FRt4/assignment_loss‚3.69106701867921 2loss/SRqqt2/assignment_loss‚1.6583719161840587 1loss/FBt3/assignment_loss‚0.16822912284745448 %loss/total_loss‚39.56200735421239 2loss/SRqqt4/assignment_loss‚1.2190061818466198 1loss/FBt4/assignment_loss‚0.13779604413624233 1loss/SRqqt3/detection_loss‚1.6372647702272207 /loss/FBt4/detection_loss‚1.8262839888532956 1loss/SRqqt4/detection_loss‚1.8757335555796721 .loss/FBt2/detection_loss‚2.277262416150835 .loss/FBt3/detection_loss‚2.307608445485433 _step‚645*P¿îc¢!Ý 1loss/SRqqt4/detection_loss‚1.8757335555796721 .loss/FBt2/detection_loss‚2.277262416150835 .loss/FBt3/detection_loss‚2.307608445485433 _step‚645 0loss/FRt2/assignment_loss‚3.1071214520412944 epoch‚6 1loss/FBt2/assignment_loss‚0.12006868949780466 .loss/FRt2/detection_loss‚2.169061642626157 /loss/FRt4/detection_loss‚2.9435088272605623 1loss/SRqqt2/detection_loss‚2.0220147325442386 $ _timestamp‚1.788435920509428e+09 /loss/FRt3/assignment_loss‚2.800451846349807 2loss/SRqqt1/assignment_loss‚0.7817445444719245 2loss/SRqqt3/assignment_loss‚1.2716626702403684 /loss/FRt1/detection_loss‚1.7168301933014125 1loss/SRqqt1/detection_loss‚0.9832073503972164 trainer/global_step‚7849 _runtime‚7898.864426963 3loss/FBt1/assignment_loss‚0.0023255506431950294 /loss/FRt3/detection_loss‚2.4591737056062337 /loss/FBt1/detection_loss‚0.6514424632973316 0loss/FRt1/assignment_loss‚1.7347702259448037 .loss/FRt4/assignment_loss‚3.69106701867921 2loss/SRqqt2/assignment_loss‚1.6583719161840587 1loss/FBt3/assignment_loss‚0.16822912284745448 %loss/total_loss‚39.56200735421239 2loss/SRqqt4/assignment_loss‚1.2190061818466198 1loss/FBt4/assignment_loss‚0.13779604413624233 1loss/SRqqt3/detection_loss‚1.6372647702272207 /loss/FBt4/detection_loss‚1.8262839888532956…–.fVž£!‚(:“ Ö»åÔ×¥ü´ gpu.0.gpu‚84.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚137.582' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.516399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ãß4w¤!‚(:ì Ö»åÔðáÖ¶ disk.dm-1.out‚ 296.6796875 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚63643.63134765625 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0# disk.dm-0.out‚63643.63134765625 disk.sda3.in‚0.203125 disk.drbd2265.in‚0 network.recv‚ 2.669379e+06 disk.sda3.out‚ 296.6796875 disk.sda1.in‚0" proc.memory.rssMB‚ 2951.4296875 disk.sda2.in‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 73.11328125 disk.rbd1.out‚ 4.134765625( disk.nvme0c0n1.out‚63643.63134765625 disk.drbd2080.in‚0 disk.sda.out‚ 296.6796875 disk.rbd0.out‚0# memory_percent‚99.5537519454956 network.sent‚ 1.9449266e+07) disk./.usagePercent‚19.08091175465352% disk./.usageGB‚284.31937408447266 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125* proc.memory.percent‚18.014097213745117 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 cpu‚25.272826112070874 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0oРî¥!‚(:’ å»åÔÀå¾µ gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚300.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.08859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥œ8Ìy¦!‚(:î å»åÔºÝã¶# disk.dm-0.out‚63725.73876953125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.90234375* proc.memory.percent‚18.014097213745117" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda.out‚ 297.31640625 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0& proc.memory.availableMB‚ 73.3671875 cpu‚25.272404825384516 disk.dm-0.in‚ 466.90234375 network.recv‚ 2.67542e+06 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚63725.73876953125 disk.dm-1.in‚0.203125" proc.memory.rssMB‚ 2951.4296875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚63725.73876953125 disk.sda2.out‚0% disk./.usageGB‚284.33081817626953 disk.sda3.out‚ 297.31640625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 297.31640625$ memory_percent‚99.55220222473145 network.sent‚ 1.9557474e+07) disk./.usagePercent‚19.08167977725624 disk.rbd1.in‚0 disk.rbd1.out‚ 4.134765625 disk.sda3.in‚0.203125‹€CJ’§!‚(:‡ ô»åԖ𿵠gpu.0.gpu‚82.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚186.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mí žw¨!‚(:ì ô»åÔœ•å¶ network.sent‚ 1.9562806e+07 network.recv‚ 2.678655e+06& disk.nvme0n1.out‚63773.24658203125 disk.sda2.in‚0# disk.dm-0.out‚63773.24658203125% disk./.usageGB‚284.35428619384766 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 297.79296875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.01438331604004 disk.drbd2265.out‚0 disk.dm-1.out‚ 297.79296875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.1669921875( disk.nvme0c0n1.out‚63773.24658203125 disk.sda3.out‚ 297.79296875# proc.memory.availableMB‚73.5625 disk.dm-0.in‚ 466.90234375" proc.memory.rssMB‚ 2951.4765625 proc.cpu.threads‚17 disk.dm-1.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.out‚0 cpu‚25.271985636429363) disk./.usagePercent‚19.08325473560689$ memory_percent‚99.55101013183594 ͹2©!- _wandbruntime‚7945  _runtime‚7945éž Çª!Á _runtime‚7945.356557513 _step‚646 & lr-AdamW/pg1‚6.618010893547393e-05 & lr-AdamW/pg2‚6.618010893547393e-05 trainer/global_step‚7899 % _timestamp‚1.7884359670013518e+09ã‰N(Ì«!Æ _step‚646 & lr-AdamW/pg1‚6.618010893547393e-05 & lr-AdamW/pg2‚6.618010893547393e-05 trainer/global_step‚7899 % _timestamp‚1.7884359670013518e+09 _runtime‚7945.356557513†‘TØ2¬!- _wandbruntime‚7945  _runtime‚7945Q[üŽ­!•P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1072710787146752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16850586716967 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141541561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170075466 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1054387784033280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16879124438528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1071235241853355 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 _step‚647 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146678105 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1089589911585280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1089586779800064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7899 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1072710760755200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚939977226902528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170052360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚916559002587648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23418171397632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62022418 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16847483248043 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚939977145850880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16879124438528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚916558974453248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1054387758605312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1089586779800064 YJDeviceStatsMo§ E²nitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170075466 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1072710787146752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84655690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788435967002255e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16876019044864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141541555 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141541555 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311617027 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146678627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84655687 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311593915 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1072710760755200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141541561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚7945.357546029 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170052360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1071238370750247 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1089589911585280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23418224314880*uä (®!šP XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62022937 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16876019044864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170052360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1071238370750247 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1089589911585280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23418224314880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1072710787146752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16850586716967 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141541561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170075466 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1054387784033280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16879124438528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1071235241853355 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 _step‚647 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146678105 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1089589911585280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1089586779800064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7899 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1072710760755200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚939977226902528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170052360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚916559002587648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23418171397632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62022418 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16847483248043 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚939977145850880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16879124438528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚916558974453248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1054387758605312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1089586779800064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170075466 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1072710787146752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84655690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788435967002255e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16876019044864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141541555 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141541555 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311617027 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146678627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84655687 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311593915 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1072710760755200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141541561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚7945.357546029 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141558839 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170096582 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1071366747647671 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1089723518000640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311655427 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1089723518000640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170073450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62030583 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23421102494720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1072842281234944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚940092568850432 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141558845 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1089720386202112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23421155398656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1072842254843392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1072842281234944 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 $ _timestamp‚1.788435967902617e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16881236765696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16878131358720 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1072842254843392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84665943 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170073450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16849592041143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170096582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16852695510275 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311655427 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141558845 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146696025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚940092487812096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311632289 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141558839 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 trainer/global_step‚7899 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311632289 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62030085 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚916671385317376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84665940 Š´óÙ±!¤N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311632289 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62030085 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚916671385317376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84665940 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141558839 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170096582 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1071366747647671 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1089723518000640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311655427 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1089723518000640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170073450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62030583 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23421102494720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1072842281234944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚940092568850432 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141558845 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1089720386202112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23421155398656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1072842254843392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1072842281234944 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 $ _timestamp‚1.788435967902617e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16881236765696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16878131358720 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1072842254843392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84665943 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170073450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16849592041143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170096582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16852695510275 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311655427 K;DeviceStatsMonitor.on_train_batch_end/acti*«L$Qve_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141558845 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146696025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚940092487812096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311632289 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141558839 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 trainer/global_step‚7899ˆjØB2²!- _wandbruntime‚7946  _runtime‚7946CMÑa³!Û /loss/FRt1/detection_loss‚1.4991810967524846 2loss/SRqqt4/assignment_loss‚1.5425908150880232 .loss/FRt4/detection_loss‚3.096087884444457 1loss/SRqqt1/detection_loss‚0.8772481455249183 /loss/FBt3/detection_loss‚2.1360721479762685 .loss/FBt4/detection_loss‚2.548156792918841 2loss/SRqqt1/assignment_loss‚0.9303840075153857 %loss/total_loss‚42.96234192784543 epoch‚6 % _timestamp‚1.7884359679047265e+09 _step‚649 0loss/FRt1/assignment_loss‚1.5093803803126018 0loss/FRt2/assignment_loss‚3.6374898219809815 /loss/FRt4/assignment_loss‚4.177195539841285 .loss/FRt2/detection_loss‚2.583590188447167 _runtime‚7946.259695066 1loss/FBt2/assignment_loss‚0.03593542681275151 1loss/FBt4/assignment_loss‚0.15116816299268976 2loss/SRqqt3/assignment_loss‚1.2903935258155284 3loss/FBt1/assignment_loss‚0.0011175513885446106 1loss/FBt3/assignment_loss‚0.10392028430280996 /loss/FRt3/detection_loss‚2.5411099851936907 1loss/SRqqt4/detection_loss‚2.8595043213471123 0loss/FRt3/assignment_loss‚2.6915412400219894 1loss/SRqqt2/detection_loss‚1.9488632487399238 1loss/SRqqt3/detection_loss‚2.1068069000283014 /loss/FBt2/detection_loss‚2.4216830858162472 trainer/global_step‚7899 2loss/SRqqt2/assignment_loss‚1.6374145004720915 /loss/FBt1/detection_loss‚0.6355068741113316»**f´!à 2loss/SRqqt4/assignment_loss‚1.5425908150880232 .loss/FRt4/detection_loss‚3.096087884444457 1loss/SRqqt1/detection_loss‚0.8772481455249183 /loss/FBt3/detection_loss‚2.1360721479762685 .loss/FBt4/detection_loss‚2.548156792918841 2loss/SRqqt1/assignment_loss‚0.9303840075153857 %loss/total_loss‚42.96234192784543 epoch‚6 % _timestamp‚1.7884359679047265e+09 _step‚649 0loss/FRt1/assignment_loss‚1.5093803803126018 0loss/FRt2/assignment_loss‚3.6374898219809815 /loss/FRt4/assignment_loss‚4.177195539841285 .loss/FRt2/detection_loss‚2.583590188447167 _runtime‚7946.259695066 1loss/FBt2/assignment_loss‚0.03593542681275151 1loss/FBt4/assignment_loss‚0.15116816299268976 2loss/SRqqt3/assignment_loss‚1.2903935258155284 3loss/FBt1/assignment_loss‚0.0011175513885446106 1loss/FBt3/assignment_loss‚0.10392028430280996 /loss/FRt3/detection_loss‚2.5411099851936907 1loss/SRqqt4/detection_loss‚2.8595043213471123 0loss/FRt3/assignment_loss‚2.6915412400219894 1loss/SRqqt2/detection_loss‚1.9488632487399238 1loss/SRqqt3/detection_loss‚2.1068069000283014 /loss/FBt2/detection_loss‚2.4216830858162472 trainer/global_step‚7899 2loss/SRqqt2/assignment_loss‚1.6374145004720915 /loss/FBt1/detection_loss‚0.6355068741113316 /loss/FRt1/detection_loss‚1.4991810967524846‰¾Ã»*“µ!‚(:ˆ ƒ¼åÔœð÷¶ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚176.142' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z2ît¶!‚(:é ƒ¼åÔË≹ disk.drbd2266.out‚0 disk.sda3.out‚ 298.546875 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 298.546875$ memory_percent‚99.55141544342041" proc.memory.rssMB‚ 2951.4765625 network.sent‚ 1.9568775e+07$ disk./.usageGB‚284.3601951599121 disk.drbd2266.in‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚63842.77783203125 cpu‚25.27155305714719 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 466.90234375 disk.sda.out‚ 298.546875 disk.drbd2265.in‚0' proc.memory.availableMB‚ 73.49609375 disk.rbd1.out‚ 4.1669921875# disk.dm-0.out‚63842.77783203125 disk.drbd2080.out‚0 disk.dm-1.in‚0.203125* disk./.usagePercent‚19.083651291277427( disk.nvme0c0n1.out‚63842.77783203125 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0) proc.memory.percent‚18.01438331604004 network.recv‚ 2.682677e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 466.90234375ï¯[û“·!‚(:ˆ ’¼åԉЪµ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚299.516' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òd õ{¸!‚(:ð ’¼åÔƒŽ· network.sent‚ 1.9677492e+07" proc.memory.rssMB‚ 2951.5703125 cpu‚25.27115010834756 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.22998046875 disk.drbd1716.in‚0 disk.sda.out‚ 298.92578125 disk.sda2.in‚0 disk.dm-1.out‚ 298.92578125 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 298.92578125$ memory_percent‚99.55201148986816& proc.memory.availableMB‚ 73.3984375 disk.rbd0.out‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0$ disk./.usageGB‚284.3960609436035& disk.nvme0n1.out‚63909.86767578125 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 network.recv‚ 2.689091e+06( disk.nvme0c0n1.out‚63909.86767578125 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0* proc.memory.percent‚18.014955520629883* disk./.usagePercent‚19.086058274114354 disk.rbd1.in‚0# disk.dm-0.out‚63909.86767578125ý­èy’¹!‚(:‡ ¡¼åÔÚ°´µ gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.897 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ºï—wº!‚(:ì ¡¼åÔƤ· disk.sda1.in‚0$ memory_percent‚99.55229759216309% disk./.usageGB‚284.45311737060547 disk.dm-0.in‚ 466.90234375! disk.dm-0.out‚63943.357421875 disk.sda.in‚0.203125 disk.sda2.in‚0# proc.memory.rssMB‚ 2951.62109375 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.sda.out‚ 299.74609375 disk.rbd0.in‚0 network.sent‚ 1.9683834e+07 network.recv‚ 2.693922e+06& disk.nvme0c0n1.out‚63943.357421875 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 4.22998046875* proc.memory.percent‚18.015265464782715 cpu‚25.27073469842949 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375& proc.memory.availableMB‚ 73.3515625" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 299.74609375 disk.drbd2080.out‚0 disk.dm-1.out‚ 299.74609375 proc.cpu.threads‚17* disk./.usagePercent‚19.089887378803986$ disk.nvme0n1.out‚63943.357421875 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚09Š˜82»!- _wandbruntime‚7992  _runtime‚7992»r©Å¼!¿ trainer/global_step‚7949 % _timestamp‚1.7884360138111107e+09 _runtime‚7992.166144434 _step‚650 % lr-AdamW/pg1‚6.48023657012628e-05 % lr-AdamW/pg2‚6.48023657012628e-050*©>ʽ!Ä _step‚650 % lr-AdamW/pg1‚6.48023657012628e-05 % lr-AdamW/pg2‚6.48023657012628e-05 trainer/global_step‚7949 % _timestamp‚1.7884360138111107e+09 _runtime‚7992.166144434Š¥3ç2¾!- _wandbruntime‚7992  _runtime‚7992VÏ ª(¿!–P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1079285465164800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16981649151488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85168337 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142405761 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1096270246101504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313539079 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚922179517659648 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1096267114316288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚7949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16952936494251 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚945744643140608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1096270246101504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171133318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85168340 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62403868 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1096267114316288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1060857634094080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1079285491556352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62404409 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1077813674057255 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1077810545160363 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚922179545794048 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1060857608666112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142405755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16984754545152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171133318 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313515967 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚651 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16984754545152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1079285465164800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚945744726289408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171110212 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚7992.167134012 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313539079 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚922179517659648 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1096267114316288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚7949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16952936494251 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚945744643140608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1096270246101504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171133318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85168340 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62403868 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1096267114316288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1060857634094080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1079285491556352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62404409 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1077813674057255 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1077810545160363 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚922179545794048 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1060857608666112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142405755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16984754545152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171133318 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313515967 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚651 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16984754545152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1079285465164800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚945744726289408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171110212 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚7992.167134012 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1079416985644544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16986866872320 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23568058142720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1077942050954679 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚s/Im85178593 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85178590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚945860067704832 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1060987005667328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23568111046656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142423045 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16986866872320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16955045287351 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1096400720718336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16983761465344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚922291956658176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313554341 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147590178 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147590691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142423039 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 trainer/global_step‚7949 _step‚652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚945859986666496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313577479 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313577479 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142423039 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217HxÄ(*'Ã!¤N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16986866872320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16955045287351 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1096400720718336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16983761465344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚922291956658176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313554341 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147590178 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147590691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142423039 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 trainer/global_step‚7949 _step‚652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚945859986666496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313577479 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313577479 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142423039 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1077945179851779 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1079416985644544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16986866872320 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23568058142720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1077942050954679 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85178593 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85178590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚945860067704832 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1060987005667328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23568111046656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142423045 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17058389740459 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315438019 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1085860195965952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148467642 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85680987 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172168064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17061493209383 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1102947448832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1102950580617728 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17090384651776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚927800089000448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚655 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1085860169574400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1084385848467371 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚951512061314560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1102947448832512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1067327458726912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1085860195965952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23711972314112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23711919396864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62787191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 E;DeviceStatsMonitor.on_train_batch_¸åÁµ4start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143269955 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85680990 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1102950580617728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17090384651776 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315461131 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1084388977364263 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1102947448832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1102950580617728 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17090384651776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚927800089000448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _step‚655 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1085860169574400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1084385848467371 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚951512061314560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1102947448832512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1067327458726912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1085860195965952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23711972314112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23711919396864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62787191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143269955 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85680990 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1102950580617728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17090384651776 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315461131 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1084388977364263 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17058389740459 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315438019 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1085860195965952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148467642 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85680987 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172168064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17061493209383iá*6“Ó!‚(:ˆ ݼåÔÑʶ gpu.0.gpu‚65.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚295.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨MlÔ!‚(:á ݼåÔõ夷 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 301.84375 network.recv‚ 2.712589e+06 disk.sda3.out‚ 301.84375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚284.5572929382324 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 proc.cpu.threads‚17 network.sent‚ 1.9810505e+07 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.out‚0 disk.dm-1.out‚ 301.84375$ memory_percent‚99.55246448516846 disk.sda2.in‚0 disk.rbd1.out‚ 4.29296875 disk.dm-0.in‚ 466.90234375# proc.memory.rssMB‚ 2951.62109375* proc.memory.percent‚18.015265464782715 cpu‚25.269061555541253* disk./.usagePercent‚19.096878688556554 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 73.32421875 disk.rbd1.in‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚64191.669921875& disk.nvme0c0n1.out‚64191.669921875! disk.dm-0.out‚64191.669921875l™I_2Õ!- _wandbruntime‚8040  _runtime‚8040X`8&'Ö! N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17089391571968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315499531 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143287239 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172189154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172212286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1103081055234560 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143287245 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚951627327620608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148485446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚927912499864576 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1067456881156096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143287245 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315499531 % _timestamp‚1.7884360619720774e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085991690054144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17060498533559 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1103084187033088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62794723 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315476393 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23714908794368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17092496978944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85691240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1085991690054144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1085991663662592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17063602002691 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143287239 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172212286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62794206 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17092496978944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚7999<´¦~#×!¥N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17092496978944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚7999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17089391571968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315499531 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143287239 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172189154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172212286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1103081055234560 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143287245 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚951627327620608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148485446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚927912499864576 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1067456881156096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143287245 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315499531 % _timestamp‚1.7884360619720774e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085991690054144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17060498533559 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1103084187033088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62794723 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315476393 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23714908794368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17092496978944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85691240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocation¼0” s.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1085991690054144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1085991663662592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17063602002691 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143287239 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172212286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62794206 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317360071 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86193640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8086.884009667 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1109630915133952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚933420604072448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173225916 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚933420632206848 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1090964280671271 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17163842986667 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63167524 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23858538706944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚8049 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1109627783348736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1073797308787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1092434873984000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149361164 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1092434900375552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302ò5~!(æ!›P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1092434900375552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1109630915133952 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1092434873984000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317383183 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 % _timestamp‚1.7884361085287726e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1092434900375552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚957279170913792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144134155 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86193637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173249022 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚957279087764992 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1073797334215680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144134161 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173225916 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63166981 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317360071 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86193640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8086.884009667 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1109630915133952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚933420604072448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173225916 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚933420632206848 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1090964280671271 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17163842986667 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63167524 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23858538706944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚8049 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1109627783348736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1073797308787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1092434873984000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149361164 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0“<Ò§ 2ç!- _wandbruntime‚8087  _runtime‚8087…ÄÃíè! N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1092566368072192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1109761389750784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1092566394463744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚957394500657664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317398445 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173247006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 XDDeviceStatsMonitor.on_train_bmž¦9"atch_end/inactive_split_bytes.all.freed‚957394419619328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17198127085568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 trainer/global_step‚8049 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884361094266121e+09 _runtime‚8087.782440589 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173247006 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317421583 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144151445 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17169055248899 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149378403 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23861404682752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144151439 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63174513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚933533014936576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23861457586688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144151445 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249âÝì+'é!¥N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1092566368072192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1109761389750784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1092566394463744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚957394500657664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317398445 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173247006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚957394419619328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17198127085568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 trainer/global_step‚8049 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884361094266121e+09 _runtime‚8087.782440589 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173247006 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317421583 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144151445 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17169055248899 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149378403 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23861404682752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144151439 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63174513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚933533014936576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23861457586688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144151445 E²ÍÌõ!Æ & lr-AdamW/pg2‚6.071714449380056e-05 trainer/global_step‚8099 % _timestamp‚1.7884361560217428e+09 _runtime‚8134.376775713 _step‚662 & lr-AdamW/pg1‚6.071714449380056e-05–‹a†2ö!- _wandbruntime‚8134  _runtime‚8134uQmD:÷!–P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319282123 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1099009604785152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144998355 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63549021 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚963045907517952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1116308117864960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144998361 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17301644865024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319305235 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚8099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17298539471360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚8134.377790528 _step‚663 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174306874 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63549562 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1097536455081387 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1080267184276480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.s€òÑÍâmall_pool.freed‚ 174283768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1080267158848512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17301644865024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86706290 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚939041147278848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17298539471360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174283768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1097539583978279 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150255852 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144998355 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86706287 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17272399701799 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1116308117864960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144998361 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17301644865024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319305235 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚8099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17298539471360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚8134.377790528 _step‚663 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174306874 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63549562 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1097536455081387 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1080267184276480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174283768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1080267158848512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17301644865024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86706290 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚939041147278848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17298539471360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174283768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1097539583978279 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150255852 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144998355 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86706287 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17272399701799 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17300651785216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚963161264840192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1116444856065536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1099141098873344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1099141072481792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63556642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1116444856065536 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174327990 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1097667960875703 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939153586277376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145015639 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319320497 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150273705 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚963161347975680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145015645 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1099141098873344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145015639 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884361569204524e+09 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1080396555849728 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174304858 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1099141072481792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24007706697216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1080396581277696 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86716540 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939153558142976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319343635 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319320497 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150273182 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24007761698304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1097671089772803 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17300651785216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17274508495107 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17271405025975 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319343635 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319320497 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150273182 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24007761698304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1097671089772803 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17300651785216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17274508495107 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17271405025975 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17300651785216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚963161264840192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1116444856065536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1099141098873344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1099141072481792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63556642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1116444856065536 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174327990 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1097667960875703 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939153586277376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsl~H©Monitor.on_train_batch_end/active.large_pool.freed‚ 145015639 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319320497 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150273705 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚963161347975680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145015645 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1099141098873344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145015639 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884361569204524e+09 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1080396555849728 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174304858 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1099141072481792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24007706697216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1080396581277696 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217˜"_Tð2ü!- _wandbruntime‚8135  _runtime‚8135à ß\ý!Ö /loss/FBt2/detection_loss‚1.9641437232494354 &loss/total_loss‚40.932192857464926 _runtime‚8135.277594546 2loss/SRqqt2/assignment_loss‚1.2336978221481496 /loss/FRt1/detection_loss‚1.4594017595438094 % _timestamp‚1.7884361569225976e+09 /loss/FRt3/assignment_loss‚2.372097360152824 2loss/SRqqt3/assignment_loss‚1.1789382715921914 0loss/FBt4/assignment_loss‚0.3787012550358971 1loss/SRqqt2/detection_loss‚2.4363087180889016 trainer/global_step‚8099 /loss/FRt2/assignment_loss‚3.808142167149168 /loss/FRt4/assignment_loss‚3.592460828668931 .loss/FRt2/detection_loss‚2.521665685104601 0loss/SRqqt4/detection_loss‚2.229432892508623 epoch‚6 2loss/SRqqt1/assignment_loss‚1.0841755094893633 4loss/FBt1/assignment_loss‚4.0573848808289625e-05 1loss/FBt2/assignment_loss‚0.36767783548002625 .loss/FRt3/detection_loss‚2.344163933805391 .loss/FBt3/detection_loss‚1.992993631146171 .loss/FBt4/detection_loss‚2.156556679142846 0loss/FBt3/assignment_loss‚0.3283476694806185 2loss/SRqqt4/assignment_loss‚1.4409309254913796 1loss/SRqqt1/detection_loss‚0.7452827759885362 /loss/FBt1/detection_loss‚0.6123798615764826 _step‚665 .loss/FRt4/detection_loss‚3.675521442118813 1loss/SRqqt3/detection_loss‚1.7792936411865972 0loss/FRt1/assignment_loss‚1.2298378954673637=\aþ!Û % _timestamp‚1.7884361569225976e+09 /loss/FRt3/assignment_loss‚2.372097360152824 2loss/SRqqt3/assignment_loss‚1.1789382715921914 0loss/FBt4/assignment_loss‚0.3787012550358971 1loss/SRqqt2/detection_loss‚2.4363087180889016 trainer/global_step‚8099 /loss/FRt2/assignment_loss‚3.808142167149168 /loss/FRt4/assignment_loss‚3.592460828668931 .loss/FRt2/detection_loss‚2.521665685104601 0loss/SRqqt4/detection_loss‚2.229432892508623 epoch‚6 2loss/SRqqt1/assignment_loss‚1.0841755094893633 4loss/FBt1/assignment_loss‚4.0573848808289625e-05 1loss/FBt2/assignment_loss‚0.36767783548002625 .loss/FRt3/detection_loss‚2.344163933805391 .loss/FBt3/detection_loss‚1.992993631146171 .loss/FBt4/detection_loss‚2.156556679142846 0loss/FBt3/assignment_loss‚0.3283476694806185 2loss/SRqqt4/assignment_loss‚1.4409309254913796 1loss/SRqqt1/detection_loss‚0.7452827759885362 /loss/FBt1/detection_loss‚0.6123798615764826 _step‚665 .loss/FRt4/detection_loss‚3.675521442118813 1loss/SRqqt3/detection_loss‚1.7792936411865972 0loss/FRt1/assignment_loss‚1.2298378954673637 /loss/FBt2/detection_loss‚1.9641437232494354 &loss/total_loss‚40.932192857464926 _runtime‚8135.277594546 2loss/SRqqt2/assignment_loss‚1.2336978221481496 /loss/FRt1/detection_loss‚1.4594017595438094™Ô±¼žÿ!‚(:“ ƽåԴ•µ gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.784' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.956799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E*æs€"‚(:è ƽåÔŠ†ø¶ disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 proc.cpu.threads‚17* disk./.usagePercent‚19.100100799382503 disk.dm-1.out‚ 305.703125 disk.sda3.out‚ 305.703125 disk.sda.in‚0.203125 disk.rbd0.in‚0 cpu‚25.266218156834487 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 305.703125 disk.drbd2265.out‚0 network.sent‚ 2.0137869e+07 disk.rbd1.out‚ 4.29296875" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.rbd0.out‚0$ disk./.usageGB‚284.6053047180176 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚64572.28173828125 disk.sda2.out‚0( proc.memory.percent‚18.0161714553833 network.recv‚ 2.747719e+06 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0' proc.memory.availableMB‚ 72.68359375# proc.memory.rssMB‚ 2951.76953125 disk.sda1.in‚0 disk.drbd2080.in‚0$ memory_percent‚99.55637454986572 disk.drbd2266.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚64572.28173828125# disk.dm-0.out‚64572.28173828125 disk.sda3.in‚0.2031259órÖ“"‚(:ˆ Õ½åÔĮٴ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚303.818' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.7636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÍŒs‚"‚(:è Õ½åÔŒ—·* proc.memory.percent‚18.016457557678223 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.0246952e+07( disk.nvme0c0n1.out‚64615.73486328125# disk.dm-0.out‚64615.73486328125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 306.421875 disk.drbd2266.out‚0* disk./.usagePercent‚19.100943320177684% disk./.usageGB‚284.61785888671875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 306.421875 disk.sda.in‚0.203125 network.recv‚ 2.75442e+06 disk.rbd1.out‚ 4.29296875 disk.drbd1716.out‚0 disk.sda.out‚ 306.421875 disk.rbd0.in‚0$ memory_percent‚99.55816268920898# proc.memory.rssMB‚ 2951.81640625 cpu‚25.265820976289838 proc.cpu.threads‚17 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚64615.73486328125 disk.dm-1.in‚0.203125% proc.memory.availableMB‚ 72.390625‹ÝOƒ"‚(:’ ä½åÔ©Ž¨µ gpu.0.gpu‚84.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.789' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.55779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â¦Èw„"‚(:ì ä½åÔâ”·* disk./.usagePercent‚19.102396674949567 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 306.83984375) proc.memory.percent‚18.01743507385254$ disk./.usageGB‚284.6395149230957 disk.rbd1.out‚ 4.29296875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.out‚0# disk.dm-0.out‚64663.74853515625 disk.sda.out‚ 306.83984375 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.90234375 disk.drbd2080.out‚0 disk.drbd2265.out‚0 network.sent‚ 2.0252242e+07 network.recv‚ 2.757698e+06 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚64663.74853515625 disk.sda1.out‚0& disk.nvme0n1.out‚64663.74853515625 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2080.in‚0$ memory_percent‚99.55763816833496& proc.memory.availableMB‚ 72.4765625 cpu‚25.26539526908809 disk.dm-1.out‚ 306.83984375 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2951.9765625 proc.cpu.threads‚17¤L|2…"- _wandbruntime‚8181  _runtime‚8181·øOôdž"Á & lr-AdamW/pg1‚5.937235287847133e-05 & lr-AdamW/pg2‚5.937235287847133e-05 trainer/global_step‚8149 % _timestamp‚1.7884362034904323e+09 _runtime‚8181.845497878 _step‚666ƒiYṡ"Æ & lr-AdamW/pg1‚5.937235287847133e-05 & lr-AdamW/pg2‚5.937235287847133e-05 trainer/global_step‚8149 % _timestamp‚1.7884362034904323e+09 _runtime‚8181.845497878 _step‚666š&¸0o2ˆ"- _wandbruntime‚8181  _runtime‚8181!+_â(‰"•P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1086737034337280 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1105584282803200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1105584282803200 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145862561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚968813358730752 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122991584166400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚944661718619648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24151585096704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17377852948007 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63930712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175364726 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24151640111104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145862555 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122988452381184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17404169577984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1105584309194752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63931259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚667 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17374749479083 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17404169577984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151150199 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17407274971648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145862561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87218940 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321227287 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175364726 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151149649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87218937 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 8181.84654236 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122988452381184 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17407274971648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175341620 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1122991584166400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚968813275581952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1104114887285287 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1104111758388395 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1086737008909312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884362034912136e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175341620 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚944661690485248Ô³’ (Š"šP M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145862555 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122988452381184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17404169577984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1105584309194752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63931259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚667 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17374749479083 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17404169577984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151150199 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17407274971648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145862561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87218940 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321227287 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175364726 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151149649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87218937 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 8181.84654236 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122988452381184 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17407274971648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175341620 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1122991584166400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚968813275581952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1104114887285287 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1104111758388395 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1086737008909312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884362034912136e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175341620 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚944661690485248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1086737034337280 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1105584282803200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1105584282803200 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145862561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚968813358730752 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122991584166400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚944661718619648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24151585096704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17377852948007 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63930712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175364726 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24151640111104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0›"Ó 2‹"- _wandbruntime‚8182  _runtime‚8182äë[Œ"žN YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17406281891840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145879839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63938654 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ]JDeviceStatsMonitor.on_Ì­¤fÉtrain_batch_end/requested_bytes.small_pool.allocated‚17379961741315 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105715803282944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145879845 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87229190 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175385842 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24154549360640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1105715776891392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17409387298816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321265687 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175362710 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1086866405910528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚944774101349376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145879845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1086866431338496 _step‚668 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1104246393079811 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚968928595708928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1123122058783232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175385842 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151167847 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1105715776891392 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 % _timestamp‚1.7884362043844905e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145879839 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151167324 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1104243264182711 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17376858272183 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚ 8182.7402691 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321242549 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175362710 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24154494359552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1123125190581760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17406281891840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17409387298816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚944774129483776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151167324 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1104243264182711 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17376858272183 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚ 8182.7402691 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321242549 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175362710 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24154494359552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1123125190581760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17406281891840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17409387298816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚944774129483776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145879839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63938654 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17379961741315 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105715803282944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145879845 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87229190 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175385842 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24154549360640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1105715776891392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17409387298816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321265687 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175362710 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1086866405910528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚944774101349376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145879845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1086866431338496 _step‚668 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1104246393079811 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚968928595708928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456œ¾J¢Ì2Ž"- _wandbruntime‚8182  _runtime‚8182øÐß“V"Ð .loss/FRt2/detection_loss‚2.439156336443768 /loss/FBt1/detection_loss‚0.6369380175873831 /loss/FRt3/assignment_loss‚2.323412299156189 2loss/SRqqt1/assignment_loss‚0.9209738563280553 1loss/SRqqt4/detection_loss‚2.2245348726000103 epoch‚6 1loss/SRqqt1/detection_loss‚0.8518360636197031 $ _timestamp‚1.788436204386426e+09 /loss/FRt2/assignment_loss‚3.405828779084342 0loss/SRqqt4/assignment_loss‚1.22062686628529 -loss/FRt1/detection_loss‚1.61400204601095 1loss/SRqqt2/assignment_loss‚1.512279410140984 2loss/SRqqt3/assignment_loss‚1.2669748967608152 0loss/FBt2/assignment_loss‚0.3602715798964103 .loss/FRt3/detection_loss‚2.509787733807708 _step‚669 0loss/FRt1/assignment_loss‚1.2802297589825649 0loss/FBt3/assignment_loss‚0.1511406931954711 1loss/SRqqt2/detection_loss‚2.2483009614727716 /loss/FBt2/detection_loss‚1.9403742949167888 /loss/FBt3/detection_loss‚2.5017887462269175 /loss/FBt4/detection_loss‚2.8354260325431824 trainer/global_step‚8149 /loss/FRt4/assignment_loss‚3.244838819359288 0loss/FBt4/assignment_loss‚0.5197799316208277 /loss/FRt4/detection_loss‚2.7697289820873374 %loss/total_loss‚40.50278578388815 2loss/FBt1/assignment_loss‚0.006504206915747902 1loss/SRqqt3/detection_loss‚1.7180505988456436 _runtime‚8182.741393893=•wÇ["Õ /loss/FBt3/detection_loss‚2.5017887462269175 /loss/FBt4/detection_loss‚2.8354260325431824 trainer/global_step‚8149 /loss/FRt4/assignment_loss‚3.244838819359288 0loss/FBt4/assignment_loss‚0.5197799316208277 /loss/FRt4/detection_loss‚2.7697289820873374 %loss/total_loss‚40.50278578388815 2loss/FBt1/assignment_loss‚0.006504206915747902 1loss/SRqqt3/detection_loss‚1.7180505988456436 _runtime‚8182.741393893 .loss/FRt2/detection_loss‚2.439156336443768 /loss/FBt1/detection_loss‚0.6369380175873831 /loss/FRt3/assignment_loss‚2.323412299156189 2loss/SRqqt1/assignment_loss‚0.9209738563280553 1loss/SRqqt4/detection_loss‚2.2245348726000103 epoch‚6 1loss/SRqqt1/detection_loss‚0.8518360636197031 $ _timestamp‚1.788436204386426e+09 /loss/FRt2/assignment_loss‚3.405828779084342 0loss/SRqqt4/assignment_loss‚1.22062686628529 -loss/FRt1/detection_loss‚1.61400204601095 1loss/SRqqt2/assignment_loss‚1.512279410140984 2loss/SRqqt3/assignment_loss‚1.2669748967608152 0loss/FBt2/assignment_loss‚0.3602715798964103 .loss/FRt3/detection_loss‚2.509787733807708 _step‚669 0loss/FRt1/assignment_loss‚1.2802297589825649 0loss/FBt3/assignment_loss‚0.1511406931954711 1loss/SRqqt2/detection_loss‚2.2483009614727716 /loss/FBt2/detection_loss‚1.9403742949167888œk¼h‘‘"‚(:† ó½åÔÔý›µ gpu.0.gpu‚89.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚129.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘¾w’"‚(:ì ó½åÔÌû€· disk.rbd1.in‚0( disk.nvme0c0n1.out‚64695.73291015625# disk.dm-0.out‚64695.73291015625 disk.drbd2080.in‚0 cpu‚25.26500297335972 network.sent‚ 2.025826e+07 network.recv‚ 2.762262e+06* disk./.usagePercent‚19.103617318872825 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda.out‚ 307.55078125! proc.memory.rssMB‚ 2952.140625 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚64695.73291015625 disk.sda3.out‚ 307.55078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚284.6577033996582 disk.dm-1.out‚ 307.55078125 disk.dm-0.in‚ 466.90234375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125' proc.memory.availableMB‚ 72.01953125 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.29296875$ memory_percent‚99.56042766571045* proc.memory.percent‚18.018436431884766áæñfž“"‚(:“ ‚¾åÔÿͳµ gpu.0.gpu‚55.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚274.822' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.964400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x?n”"‚(:ã ‚¾åÔ”ŒÙ¶% disk.nvme0n1.out‚64761.2705078125 disk.rbd1.out‚ 4.29296875 disk.sda3.out‚ 307.921875! proc.memory.rssMB‚ 2952.140625* proc.memory.percent‚18.018436431884766* disk./.usagePercent‚19.104791369424852 disk.rbd1.in‚0 cpu‚25.26461313835969 disk.drbd2265.out‚0 network.sent‚ 2.0367e+07 network.recv‚ 2.768395e+06% disk./.usageGB‚284.67519760131836 disk.sda1.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 307.921875 disk.sda3.in‚0.203125$ memory_percent‚98.76387119293213 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.dm-0.out‚64761.2705078125( proc.memory.availableMB‚ 202.52734375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚64761.2705078125 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda.out‚ 307.921875 disk.drbd2080.out‚0 disk.sda2.out‚0³¬ž•"‚(:“ ‘¾åÔž—µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚308.951' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.790200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eö9v–"‚(:ë ‘¾åÔªÿ©· network.recv‚ 2.773286e+06$ disk./.usageGB‚284.6861267089844 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125" disk.dm-0.out‚64805.6923828125 disk.drbd2266.out‚0 disk.sda3.out‚ 308.48828125$ memory_percent‚98.76387119293213( proc.memory.availableMB‚ 202.52734375! proc.memory.rssMB‚ 2952.140625 disk.dm-0.in‚ 466.90234375 network.sent‚ 2.0373396e+07 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 308.48828125 disk.rbd1.out‚ 4.29296875 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0* proc.memory.percent‚18.018436431884766 disk.rbd0.in‚0% disk.nvme0n1.out‚64805.6923828125 disk.drbd2266.in‚0 cpu‚25.26424898711413 proc.cpu.threads‚17* disk./.usagePercent‚19.105524831010452 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 308.48828125' disk.nvme0c0n1.out‚64805.6923828125 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375¦eåß2—"- _wandbruntime‚8229  _runtime‚8229&ÑƘ"À _step‚670 & lr-AdamW/pg1‚5.803650423893487e-05 & lr-AdamW/pg2‚5.803650423893487e-05 trainer/global_step‚8199 $ _timestamp‚1.788436251203155e+09 _runtime‚8229.558183575ÙðË™"Å $ _timestamp‚1.788436251203155e+09 _runtime‚8229.558183575 _step‚670 & lr-AdamW/pg1‚5.803650423893487e-05 & lr-AdamW/pg2‚5.803650423893487e-05 trainer/global_step‚8199ž$‘©¡2š"- _wandbruntime‚8229  _runtime‚8229 „âàž›"•P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1129668786897408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17483306194215 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87731590 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64310344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1129668786897408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152041371 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1129671918682624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17509799684608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17512905078272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 trainer/global_step‚8199 _runtime‚8229.559275607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152041934 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1112158987212800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146726761 ZEDeviceStatsMonitor.on_traiS̼/} n_batch_start/requested_bytes.all.allocated‚1110690190592295 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64310344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1129668786897408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152041371 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1129671918682624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17509799684608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17512905078272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 trainer/global_step‚8199 _runtime‚8229.559275607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152041934 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1112158987212800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146726761 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1110690190592295 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24300621460480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1110818567489719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87741840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24300678558720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17515017405440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚974695266016256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1093336281399296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1112290507692544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64317267 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146744045 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87741843 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176443694 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152059649 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1110821696386819 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1112290481300992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _runtime‚8230.455927653 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1129805525097984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚950394672690176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17511911998464 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1112290507692544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64317806 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176420562 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323164601 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17485414987523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚974695351248896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146744039 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152059107 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17515017405440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146744039 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚950394644555776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323187739 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1129805525097984 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176443694 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323187739 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302ùe}é"Ÿ"¤N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1093336281399296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1112290507692544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64317267 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146744045 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87741843 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176443694 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152059649 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1110821696386819 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1112290481300992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _runtime‚8230.455927653 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1129805525097984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚950394672690176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17511911998464 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1112290507692544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64317806 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176420562 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323164601 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17485414987523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚974695351248896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146744039 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152059107 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17515017405440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146744039 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚950394644555776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323187739 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1129805525097984 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176443694 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323187739 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 trainer/global_step‚8199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24300621460480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1110818567489719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87741840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24300678558720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17515017405440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚974695266016256 !Ÿ°2 "- _wandbruntime‚8230  _runtime‚8230Bi{#Z¡"Ô trainer/global_step‚8199 _step‚673 1loss/FBt4/assignment_loss‚0.13670162067071281 .loss/FRt4/detection_loss‚2.643490625394357 /loss/FBt2/detection_loss‚1.9937942624092102 /loss/FBt4/detection_loss‚2.5327750661156396 _runtime‚8230.457187852 0loss/SRqqt3/detection_loss‚2.178951835259795 2loss/SRqqt4/assignment_loss‚1.9091080660621325 1loss/SRqqt1/detection_loss‚0.8494558110833168 0loss/SRqqt4/detection_loss‚2.435704642534256 .loss/FBt3/detection_loss‚2.051320751508077 0loss/FRt3/assignment_loss‚2.9305937505430646 /loss/FRt4/assignment_loss‚3.043011407594423 2loss/SRqqt1/assignment_loss‚0.9726256410386868 2loss/FBt1/assignment_loss‚0.013152970521698901 $ _timestamp‚1.788436252102183e+09 epoch‚6 0loss/FRt2/assignment_loss‚3.2649052954382367 0loss/FBt2/assignment_loss‚0.4115861138488981 .loss/FRt2/detection_loss‚2.450925878352589 &loss/total_loss‚41.019197900287764 0loss/FBt3/assignment_loss‚0.1697394703126823 /loss/FRt1/detection_loss‚1.6903781737793575 1loss/SRqqt2/detection_loss‚2.2352609112858772 /loss/FBt1/detection_loss‚0.6378590490177765 0loss/FRt1/assignment_loss‚1.4257570158351551 1loss/SRqqt2/assignment_loss‚1.318353159003891 .loss/FRt3/detection_loss‚2.458457924425602 2loss/SRqqt3/assignment_loss‚1.2652884582523256•ˆØ_¢"Ù _step‚673 1loss/FBt4/assignment_loss‚0.13670162067071281 .loss/FRt4/detection_loss‚2.643490625394357 /loss/FBt2/detection_loss‚1.9937942624092102 /loss/FBt4/detection_loss‚2.5327750661156396 _runtime‚8230.457187852 0loss/SRqqt3/detection_loss‚2.178951835259795 2loss/SRqqt4/assignment_loss‚1.9091080660621325 1loss/SRqqt1/detection_loss‚0.8494558110833168 0loss/SRqqt4/detection_loss‚2.435704642534256 .loss/FBt3/detection_loss‚2.051320751508077 0loss/FRt3/assignment_loss‚2.9305937505430646 /loss/FRt4/assignment_loss‚3.043011407594423 2loss/SRqqt1/assignment_loss‚0.9726256410386868 2loss/FBt1/assignment_loss‚0.013152970521698901 $ _timestamp‚1.788436252102183e+09 epoch‚6 0loss/FRt2/assignment_loss‚3.2649052954382367 0loss/FBt2/assignment_loss‚0.4115861138488981 .loss/FRt2/detection_loss‚2.450925878352589 &loss/total_loss‚41.019197900287764 0loss/FBt3/assignment_loss‚0.1697394703126823 /loss/FRt1/detection_loss‚1.6903781737793575 1loss/SRqqt2/detection_loss‚2.2352609112858772 /loss/FBt1/detection_loss‚0.6378590490177765 0loss/FRt1/assignment_loss‚1.4257570158351551 1loss/SRqqt2/assignment_loss‚1.318353159003891 .loss/FRt3/detection_loss‚2.458457924425602 2loss/SRqqt3/assignment_loss‚1.2652884582523256 trainer/global_step‚8199¡ùÖŸ“£"‚(:ˆ  ¾åÔ÷©µ gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚283.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jY#y¤"‚(:î  ¾åÔ³»‘· disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375" disk.nvme0c0n1.in‚ 466.90234375 disk.rbd0.in‚0 disk.dm-1.out‚ 309.09765625 disk.sda3.in‚0.203125 disk.sda1.out‚0! proc.memory.rssMB‚ 2952.140625* disk./.usagePercent‚19.106379384159744 disk.drbd1716.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚18.018436431884766 network.sent‚ 2.037851e+07 network.recv‚ 2.775966e+06& disk.nvme0n1.out‚64847.90478515625( disk.nvme0c0n1.out‚64847.90478515625 disk.dm-0.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 309.09765625( proc.memory.availableMB‚ 202.71484375 cpu‚25.26383092188368 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.29296875% disk./.usageGB‚284.69886016845703 disk.sda2.out‚0 disk.drbd2266.out‚0# disk.dm-0.out‚64847.90478515625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 309.09765625 disk.sda1.in‚0$ memory_percent‚98.76272678375244ø²Ÿqž¥"‚(:“ ¯¾åÔò³ÿ´ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t›ñsy¦"‚(:î ¯¾åÔÖ‘æ·$ disk./.usageGB‚284.6971244812012 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2952.140625 disk.sda3.in‚0.203125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 466.90234375 disk.drbd1716.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚64892.25634765625 disk.sda2.in‚0 disk.dm-1.out‚ 309.68359375* proc.memory.percent‚18.018436431884766 network.recv‚ 2.782379e+06* disk./.usagePercent‚19.106262900731664# disk.dm-0.out‚64892.25634765625 disk.sda1.in‚0 disk.dm-1.in‚0.203125 cpu‚25.263469417147633 disk.rbd1.out‚ 4.29296875 disk.sda3.out‚ 309.68359375 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2266.in‚0' proc.memory.availableMB‚ 202.7109375" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.sent‚ 2.0486651e+07( disk.nvme0c0n1.out‚64892.25634765625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375$ memory_percent‚98.76275062561035 disk.sda.out‚ 309.68359375 disk.sda2.out‚0{©W^œ§"‚(:‘ ¾¾åÔ ‹µ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚205.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.150200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôx´n¨"‚(:ã ¾¾åÔ÷Âù· disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.sda.out‚ 310.15625( proc.memory.availableMB‚ 202.71484375! proc.memory.rssMB‚ 2952.140625' disk.nvme0c0n1.out‚64951.3916015625 network.recv‚ 2.786747e+06) disk./.usagePercent‚19.10861356191106 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 310.15625 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 network.sent‚ 2.0492646e+07% disk./.usageGB‚284.73215103149414 disk.sda2.out‚0" disk.dm-0.out‚64951.3916015625 disk.drbd1716.out‚0 disk.rbd0.out‚0$ memory_percent‚98.76272678375244 cpu‚25.263052090581702" disk.nvme0c0n1.in‚ 466.90234375 disk.nvme0n1.in‚ 466.90234375* proc.memory.percent‚18.018436431884766 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 4.29296875 disk.sda3.in‚0.203125 disk.sda3.out‚ 310.15625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125% disk.nvme0n1.out‚64951.3916015625 disk.drbd2265.in‚0 disk.rbd0.in‚0–‘•—2©"- _wandbruntime‚8277  _runtime‚8277d Ǫ"Á trainer/global_step‚8249 % _timestamp‚1.7884362987044766e+09 _runtime‚8277.059544694 _step‚674 & lr-AdamW/pg1‚5.670987696404549e-05 & lr-AdamW/pg2‚5.670987696404549e-05‘¨ä—Ì«"Æ _step‚674 & lr-AdamW/pg1‚5.670987696404549e-05 & lr-AdamW/pg2‚5.670987696404549e-05 trainer/global_step‚8249 % _timestamp‚1.7884362987044766e+09 _runtime‚8277.059544694¢ >H2¬"- _wandbruntime‚8277  _runtime‚8277”Sš(­"–P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177457324 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1136352253198848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24443873929216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚955902805032448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136349121413632 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17615429791232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17615429791232 _runtime‚8277.060476744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88244237 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177457324 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24443816817664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118733718013952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1099676709030912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1099676734458880 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147590955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884362987052195e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325048279 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147590961 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147590955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147590961 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8249 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17618535184896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88244240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1117265493899303 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177480430 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118733691622400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚566 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚675 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17588759440423 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚980346593715712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24443873929216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚955902805032448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136349121413632 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17615429791232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17615429791232 _runtime‚8277.060476744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88244237 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177457324 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24443816817664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118733718013952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1099676709030912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1099676734458880 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147590955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884362987052195e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325048279 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147590961 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147590955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147590961 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8249 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17618535184896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88244240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1117265493899303 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177480430 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118733691622400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚566 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _step‚675 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17588759440423£~§2¯"- _wandbruntime‚8277  _runtime‚8277ãv9<à °"ŸN _runtime‚ 8277.97422593 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚8249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147608245 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚980462006513664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325109791 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1118865212102144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325109791 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88254490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1118865185710592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177478414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177501546 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17617542105088 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚956015215896576 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152951181 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17620647512064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325086653 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24446790617088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64697245 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚676 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17590868233731 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177501546 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1136485859614208 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1136482727815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88254493 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1099806131460096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1118865185710592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚956015187762176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177478414 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17620647512064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17617542105088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152951738 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1118865212102144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325109791 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88254490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1118865185710592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177478414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177501546 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17617542105088 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚956015215896576 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152951181 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17620647512064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325086653 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24446790617088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64697245 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚676 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17590868233731 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177501546 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1136485859614208 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1136482727815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88254493 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1099806131460096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1118865185710592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚956015187762176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177478414 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17620647512064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17617542105088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152951738 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1118865212102144 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _step‚679 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65070754 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1143032587715072 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1125308422423552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88756890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65071309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚986113032469504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚961523320104448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178515176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1123837668309419 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153828199 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24589627119104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326970331 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17721059897856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326993443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8324.748255382 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚986112947223552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148455155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 31053b 93664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1125308422423552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178538282 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1123840797206311 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24589684230656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1143032587715072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88756887 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1106146559091712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1143029455929856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1125308422423552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178538282 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1123840797206311 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24589684230656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1143032587715072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88756887 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1106146559091712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1143029455929856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _step‚679 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65070754 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1143032587715072 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1125308422423552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88756890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65071309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚986113032469504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚961523320104448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178515176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1123837668309419 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153828199 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24589627119104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326970331 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17721059897856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326993443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8324.748255382 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚986112947223552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0§B>¾ð“Á"‚(:ˆ ú¾åÔƒô•µ gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚291.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à…*UuÂ"‚(:ê ú¾åÔÚîÕ¶( disk.nvme0c0n1.out‚65187.88916015625 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 312.1171875 disk.drbd2265.out‚0 proc.memory.rssMB‚ 2952.1875 network.recv‚ 2.805604e+06 disk.drbd2080.out‚0* proc.memory.percent‚18.018722534179688 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 466.90234375# disk.dm-0.out‚65187.88916015625 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 312.1171875 disk.drbd2266.in‚0% disk./.usageGB‚284.80993270874023 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.nvme0n1.in‚ 466.90234375 network.sent‚ 2.0619388e+07* disk./.usagePercent‚19.113833555534217 disk.sda3.in‚0.203125 disk.sda.out‚ 312.1171875 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.203125& disk.nvme0n1.out‚65187.88916015625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0$ memory_percent‚98.68252277374268( proc.memory.availableMB‚ 215.85546875 cpu‚25.26152930234679T ¯Ô2Ã"- _wandbruntime‚8325  _runtime‚8325ðiH&'Ä" N F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1143166194130432 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1143166194130432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65078258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178536266 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1125439890120192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1106275956092928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1125439916511744 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153845925 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚961635730968576 % _timestamp‚1.7884363472961643e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327031843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17693218010807 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚8299 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153845398 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24592600325632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚986228274195968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17726277618688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1123972303000835 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148472439 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327031843 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178536266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1125439916511744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88767143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88767140 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148472445 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚8325.651722403 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178559398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24592543227392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17723172211712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148472439 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17726277618688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327031843 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178536266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1125439916511744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88767143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88767140 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148472445 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚8325.651722403 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178559398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24592543227392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17723172211712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148472439 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17726277618688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1143166194130432 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1143166194130432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65078258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178536266 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1125439890120192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1106275956092928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/actÇ^€êive_bytes.large_pool.allocated‚1125439916511744 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153845925 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚961635730968576 % _timestamp‚1.7884363472961643e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327031843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17693218010807 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚8299 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153845398 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24592600325632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚986228274195968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17726277618688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1123972303000835¨È’ 2Æ"- _wandbruntime‚8325  _runtime‚8325Qv‰YÇ"Ó trainer/global_step‚8299 $ _timestamp‚1.788436347298033e+09 1loss/SRqqt1/detection_loss‚1.0133335784126025 1loss/SRqqt4/detection_loss‚2.6870123275688718 %loss/total_loss‚42.14258337005132 /loss/FRt1/detection_loss‚1.3906483972548611 /loss/FRt2/detection_loss‚2.3190477304160595 .loss/FRt3/detection_loss‚2.180660771562698 /loss/FBt1/detection_loss‚0.5606816518353298 0loss/FRt2/assignment_loss‚3.4570476576685905 0loss/FRt4/assignment_loss‚3.8003795250602392 0loss/FBt3/assignment_loss‚0.3580235110857757 _step‚681 0loss/FRt3/assignment_loss‚2.7341391203251293 0loss/SRqqt1/assignment_loss‚0.86279645877854 2loss/SRqqt3/assignment_loss‚1.5973129889873021 2loss/FBt1/assignment_loss‚0.001649846288174217 1loss/SRqqt3/detection_loss‚1.8653742811259102 /loss/FBt2/detection_loss‚2.3006565757095814 /loss/FRt1/assignment_loss‚1.645796081258191 1loss/FBt2/assignment_loss‚0.17001773083757143 1loss/FBt4/assignment_loss‚0.17112033472706875 -loss/FRt4/detection_loss‚3.16822466124659 epoch‚6 _runtime‚8325.653014001 2loss/SRqqt2/assignment_loss‚1.3890710612492903 2loss/SRqqt4/assignment_loss‚1.6591508948643292 1loss/SRqqt2/detection_loss‚2.1336011494909015 -loss/FBt3/detection_loss‚2.17204764214429 .loss/FBt4/detection_loss‚2.504789392153422.ù…—^È"Ø 0loss/SRqqt1/assignment_loss‚0.86279645877854 2loss/SRqqt3/assignment_loss‚1.5973129889873021 2loss/FBt1/assignment_loss‚0.001649846288174217 1loss/SRqqt3/detection_loss‚1.8653742811259102 /loss/FBt2/detection_loss‚2.3006565757095814 /loss/FRt1/assignment_loss‚1.645796081258191 1loss/FBt2/assignment_loss‚0.17001773083757143 1loss/FBt4/assignment_loss‚0.17112033472706875 -loss/FRt4/detection_loss‚3.16822466124659 epoch‚6 _runtime‚8325.653014001 2loss/SRqqt2/assignment_loss‚1.3890710612492903 2loss/SRqqt4/assignment_loss‚1.6591508948643292 1loss/SRqqt2/detection_loss‚2.1336011494909015 -loss/FBt3/detection_loss‚2.17204764214429 .loss/FBt4/detection_loss‚2.504789392153422 trainer/global_step‚8299 $ _timestamp‚1.788436347298033e+09 1loss/SRqqt1/detection_loss‚1.0133335784126025 1loss/SRqqt4/detection_loss‚2.6870123275688718 %loss/total_loss‚42.14258337005132 /loss/FRt1/detection_loss‚1.3906483972548611 /loss/FRt2/detection_loss‚2.3190477304160595 .loss/FRt3/detection_loss‚2.180660771562698 /loss/FBt1/detection_loss‚0.5606816518353298 0loss/FRt2/assignment_loss‚3.4570476576685905 0loss/FRt4/assignment_loss‚3.8003795250602392 0loss/FBt3/assignment_loss‚0.3580235110857757 _step‚681 0loss/FRt3/assignment_loss‚2.7341391203251293©?ˆ°“É"‚(:ˆ ‰¿åÔТµ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.321' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.2642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l°ÜtÊ"‚(:é ‰¿åÔà· disk.drbd2266.out‚0$ memory_percent‚98.68121147155762 cpu‚25.26114731144107 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 network.sent‚ 2.0717182e+07* disk./.usagePercent‚19.114618474634202% disk./.usageGB‚284.82162857055664 disk.drbd1716.in‚0 disk.sda3.out‚ 312.6796875 proc.memory.rssMB‚ 2952.1875 disk.rbd1.out‚ 4.29296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.90234375 network.recv‚ 2.812527e+06 disk.drbd2080.out‚0 disk.sda.out‚ 312.6796875" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.in‚0.203125' proc.memory.availableMB‚ 216.0703125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚65241.31787109375 disk.dm-1.out‚ 312.6796875* proc.memory.percent‚18.018722534179688# disk.dm-0.out‚65241.31787109375( disk.nvme0c0n1.out‚65241.31787109375 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 466.90234375g•`‘žË"‚(:“ ˜¿åÔ¯ô¢µ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚134.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.999599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04E‚zÌ"‚(:ï ˜¿åÔì釷) proc.memory.percent‚18.01888942718506 network.sent‚ 2.0814619e+07 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 216.04296875 disk.sda.out‚ 313.15234375 disk.drbd2265.in‚0* disk./.usagePercent‚19.115860879197868 disk.dm-0.in‚ 466.90234375 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2952.21484375 proc.cpu.threads‚17 disk.drbd2265.out‚0$ disk./.usageGB‚284.8401412963867 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.sda3.out‚ 313.15234375 disk.sda1.out‚0# disk.dm-0.out‚65291.43310546875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚65291.43310546875 disk.sda2.out‚0 disk.dm-1.out‚ 313.15234375 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚65291.43310546875 disk.sda1.in‚0 disk.drbd1716.in‚0 cpu‚25.26076966512375 network.recv‚ 2.817972e+06 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0$ memory_percent‚98.681378364562991‰žÍ"‚(:“ §¿åÔßò€µ gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚262.207' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.441399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q%Â×xÎ"‚(:í §¿åÔùýÅ· network.recv‚ 2.822523e+06 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda.out‚ 313.87890625 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2952.21484375* disk./.usagePercent‚19.117546432803305$ disk./.usageGB‚284.8652572631836 disk.dm-0.in‚ 466.90234375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125$ memory_percent‚98.68135452270508 disk.drbd2266.out‚0& proc.memory.availableMB‚ 216.046875) proc.memory.percent‚18.01888942718506& disk.nvme0n1.out‚65339.75537109375 disk.rbd1.out‚ 4.29296875# disk.dm-0.out‚65339.75537109375 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 313.87890625 disk.drbd2265.out‚0 cpu‚25.260390120089447 network.sent‚ 2.082064e+07 disk.sda2.in‚0 disk.dm-1.out‚ 313.87890625 proc.cpu.threads‚17 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚65339.75537109375 disk.drbd2265.in‚0•Ïª2Ï"- _wandbruntime‚8372  _runtime‚8372yzŽÀÈÐ" ' lr-AdamW/pg1‚5.4085390397416835e-05 ' lr-AdamW/pg2‚5.4085390397416835e-05 trainer/global_step‚8349 % _timestamp‚1.7884363937619588e+09 _runtime‚ 8372.11700945 _step‚682ÏO0€ÍÑ"Ç _step‚682 ' lr-AdamW/pg1‚5.4085390397416835e-05 ' lr-AdamW/pg2‚5.4085390397416835e-05 trainer/global_step‚8349 % _timestamp‚1.7884363937619588e+09 _runtime‚ 8372.11700945ªj¸DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65451266 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8349 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179596134 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1130416100513319 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149319361 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1131883100441600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89269537 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17799665932839 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚991879687255552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1149709790446080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 _runtime‚8372.118040957 _step‚683 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149319361 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17826690004480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚991879772501504 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89269540 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24735881056256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚967143891445248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1131883126833152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149319355 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1112616409152512 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1149712922231296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17796562463915»g û (Ô"šP P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1149712922231296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17796562463915 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65451266 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8349 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179596134 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1130416100513319 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149319361 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1131883100441600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89269537 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17799665932839 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚991879687255552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1149709790446080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 _runtime‚8372.118040957 _step‚683 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149319361 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17826690004480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚991879772501504 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89269540 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24735881056256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚967143891445248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1131883126833152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149319355 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1112616409152512 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180«Ar6D2Õ"- _wandbruntime‚8373  _runtime‚8373¹'S Ö" N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1112745806153728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328953895 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1112745831581696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328930757 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132014620921344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/iéZïnactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179594118 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65458949 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17798671257015 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132014620921344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149336645 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1130547606307843 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328930757 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149336645 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24738817434112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132014594529792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149336639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17801774726147 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8349 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154738739 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1149846528646656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1130544477410743 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚8373.036676761 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328930757 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132014620921344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179594118 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65458949 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17798671257015 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132014620921344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149336645 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1130547606307843 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328930757 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149336645 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24738817434112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132014594529792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149336639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17801774726147 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8349 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154738739 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1149846528646656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1130544477410743 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚8373.036676761 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328953895 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1112745831581696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1156390124962304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180653986 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚997646211746816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚972764434651648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24881862341120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155614720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17932320111104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180630880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _step‚687 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180653986 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180630880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89782187 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1136991403820327 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17902015710123 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330814435 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17932320111104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1119086284641280 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1156390124962304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚972764406517248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89782190 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1138457804851200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150183555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65832530 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1136988274923435 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0@LÙï!(æ"›P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17935425504768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 330837547 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24881805229568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚8420.017655407 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1156390124962304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180653986 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚997646211746816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚972764434651648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24881862341120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155614720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17932320111104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180630880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _step‚687 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180653986 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180630880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89782187 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1136991403820327 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17902015710123 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330814435 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17932320111104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1119086284641280 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1156390124962304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚972764406517248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89782190 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1138457804851200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150183555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65832530 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1136988274923435 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1138457804851200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997646296992768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150183561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150183555¯5$U2ç"- _wandbruntime‚8420  _runtime‚8420Ô¯ña&'è" N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17904124503223 _step‚688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚997761532140032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1138589325330944 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1138589298939392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150200839 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1137119780717751 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155631848 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65839940 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚997761615275520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150200839 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17907227972355 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1119215656214528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180675102 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1137122909614851 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1138589325330944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚972876817381376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1156526863162880 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1138589298939392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚972876845515776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180651970 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330875947 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17934432424960 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155632383 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17934432424960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330852809 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚8420.911625249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65839408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302ÿ0©|#é"¥N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180651970 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330875947 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17934432424960 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155632383 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17934432424960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330852809 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚8420.911625249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65839408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17904124503223 _step‚688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚997761532140032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1138589325330944 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1138589298939392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150200839 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1137119780717751 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155631848 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65839940 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚997761615275520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150200839 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17907227972355 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1119215656214528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonuÍÛ`¯itor.on_train_batch_end/active.small_pool.allocated‚ 180675102 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1137122909614851 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1138589325330944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚972876817381376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1156526863162880 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1138589298939392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚972876845515776°·ûh+2ê"- _wandbruntime‚8420  _runtime‚8420 â!Œ[ë"Õ 0loss/FRt3/assignment_loss‚2.4591786404873464 1loss/SRqqt1/detection_loss‚1.0169004874860312 .loss/FBt2/detection_loss‚2.532068333029747 epoch‚6 2loss/SRqqt3/assignment_loss‚1.0420462518989566 0loss/FBt2/assignment_loss‚0.4167598278719993 0loss/FRt1/assignment_loss‚1.6914453213865106 /loss/FRt2/detection_loss‚2.3810691795450576 .loss/FRt4/detection_loss‚2.960874811808268 /loss/FBt4/detection_loss‚2.5240944623947144 % _timestamp‚1.7884364425576777e+09 0loss/FRt2/assignment_loss‚2.9545616875303553 /loss/FRt1/detection_loss‚1.6474880224263126 trainer/global_step‚8399 0loss/SRqqt2/assignment_loss‚1.35049131396744 3loss/FBt1/assignment_loss‚0.0033958920570971904 1loss/FBt4/assignment_loss‚0.18801317705462375 1loss/SRqqt3/detection_loss‚1.6836298452246756 0loss/FRt4/assignment_loss‚3.5927072703838348 .loss/FRt3/detection_loss‚2.469114676117897 1loss/SRqqt2/detection_loss‚2.0570665849579703 /loss/FBt3/detection_loss‚2.8289761170744896 _runtime‚ 8420.91267469 1loss/SRqqt4/assignment_loss‚1.724388402990169 /loss/SRqqt4/detection_loss‚2.50580173648066 %loss/total_loss‚41.75685192247656 2loss/SRqqt1/assignment_loss‚0.9460260266099464 1loss/FBt3/assignment_loss‚0.23044595364262932 /loss/FBt1/detection_loss‚0.5503079000498272 _step‚689¹hÁ `ì"Ú trainer/global_step‚8399 0loss/SRqqt2/assignment_loss‚1.35049131396744 3loss/FBt1/assignment_loss‚0.0033958920570971904 1loss/FBt4/assignment_loss‚0.18801317705462375 1loss/SRqqt3/detection_loss‚1.6836298452246756 0loss/FRt4/assignment_loss‚3.5927072703838348 .loss/FRt3/detection_loss‚2.469114676117897 1loss/SRqqt2/detection_loss‚2.0570665849579703 /loss/FBt3/detection_loss‚2.8289761170744896 _runtime‚ 8420.91267469 1loss/SRqqt4/assignment_loss‚1.724388402990169 /loss/SRqqt4/detection_loss‚2.50580173648066 %loss/total_loss‚41.75685192247656 2loss/SRqqt1/assignment_loss‚0.9460260266099464 1loss/FBt3/assignment_loss‚0.23044595364262932 /loss/FBt1/detection_loss‚0.5503079000498272 _step‚689 0loss/FRt3/assignment_loss‚2.4591786404873464 1loss/SRqqt1/detection_loss‚1.0169004874860312 .loss/FBt2/detection_loss‚2.532068333029747 epoch‚6 2loss/SRqqt3/assignment_loss‚1.0420462518989566 0loss/FBt2/assignment_loss‚0.4167598278719993 0loss/FRt1/assignment_loss‚1.6914453213865106 /loss/FRt2/detection_loss‚2.3810691795450576 .loss/FRt4/detection_loss‚2.960874811808268 /loss/FBt4/detection_loss‚2.5240944623947144 % _timestamp‚1.7884364425576777e+09 0loss/FRt2/assignment_loss‚2.9545616875303553 /loss/FRt1/detection_loss‚1.6474880224263126±Ø|ëΓí"‚(:ˆ ã¿åÔò‰µ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚281.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ{E„rî"‚(:ç ã¿åÔÝ誷 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.29296875 disk.rbd1.in‚0 disk.sda3.out‚ 315.90625$ memory_percent‚98.68378639221191% disk./.usageGB‚284.90416717529297 disk.sda3.in‚0.203125' proc.memory.availableMB‚ 215.6484375# proc.memory.rssMB‚ 2952.21484375 network.sent‚ 2.0947172e+07 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚65536.58642578125 network.recv‚ 2.841851e+06 disk.nvme0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚65536.58642578125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚65536.58642578125 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.90234375) proc.memory.percent‚18.01888942718506 cpu‚25.258928135524766* disk./.usagePercent‚19.120157709652556 disk.drbd2080.in‚0 disk.sda.out‚ 315.90625 disk.dm-1.out‚ 315.90625 disk.sda1.in‚0}€Flï"‚(:á ò¿åÔ··¸ network.recv‚ 2.84814e+06 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚65579.02587890625 cpu‚25.258553190064713 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚65579.02587890625 disk.sda3.in‚0.203125* disk./.usagePercent‚19.120543256999124 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 215.8984375 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚65579.02587890625 disk.drbd2080.in‚0 disk.sda2.out‚0$ memory_percent‚98.68226051330566 network.sent‚ 2.1056e+07# disk./.usageGB‚284.909912109375 disk.sda.out‚ 316.34375 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125# proc.memory.rssMB‚ 2952.21484375) proc.memory.percent‚18.01888942718506 disk.drbd1716.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.dm-1.out‚ 316.34375 disk.sda3.out‚ 316.34375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 4.29296875 disk.rbd0.in‚0)œw’ð"‚(:‡ ò¿åÔÖÆ’· gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚186.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P§oœ“ñ"‚(:ˆ ÀåÔ‡³…µ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚136.761' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼þâ oò"‚(:ä ÀåÔ°Ñã·" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0% disk./.usageGB‚284.92613983154297 disk.sda.out‚ 316.90625 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125' disk.nvme0c0n1.out‚65617.7177734375" disk.dm-0.out‚65617.7177734375 disk.sda3.in‚0.203125) disk./.usagePercent‚19.12163231304978 network.recv‚ 2.852817e+06 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.dm-1.out‚ 316.90625 disk.drbd2266.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 215.64453125 cpu‚25.258186757638867 disk.rbd0.out‚0 disk.dm-0.in‚ 466.90234375$ memory_percent‚98.68381023406982 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.29296875 disk.sda3.out‚ 316.90625 disk.sda1.out‚0# proc.memory.rssMB‚ 2952.21484375) proc.memory.percent‚18.01888942718506 network.sent‚ 2.1062228e+07 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0n1.out‚65617.7177734375H—Ý”2ó"- _wandbruntime‚8467  _runtime‚8467é´÷0Çô"Á _step‚690 & lr-AdamW/pg1‚5.150108082087124e-05 & lr-AdamW/pg2‚5.150108082087124e-05 trainer/global_step‚8449 % _timestamp‚1.7884364892884743e+09 _runtime‚8467.643539506VXÛÑÌõ"Æ & lr-AdamW/pg1‚5.150108082087124e-05 & lr-AdamW/pg2‚5.150108082087124e-05 trainer/global_step‚8449 % _timestamp‚1.7884364892884743e+09 _runtime‚8467.643539506 _step‚690²ÔvK2ö"- _wandbruntime‚8467  _runtime‚8467ú¶ñÅ(÷"˜P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156507415 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1003412469197312 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1003412381854208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18010572425255 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1163073591263744 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18037950217728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25027432130560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18037950217728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332759599 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1143563578230443 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18041055611392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚583 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚978384949723648 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125556109274112 trainer/global_step‚8449 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1125556134702080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90294837 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151047761 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25027491339264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332736487 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1145032509260800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181688732 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1163070459478528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181711838 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181688732 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884364892893827e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚586 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚691 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1163070459478528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1145032509260800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1145032535652352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90294840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1145032535652352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1143566707127335 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1163073591263744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18041055611392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18007468956331 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚978384977858048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156508001 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302_?Y#(ø"P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18041055611392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18007468956331 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚978384977858048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156508001 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156507415 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1003412469197312 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1003412381854208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18010572425255 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1163073591263744 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18037950217728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25027432130560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18037950217728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332759599 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1143563578230443 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18041055611392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚583 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚978384949723648 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125556109274112 trainer/global_step‚8449 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1125556134702080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90294837 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151047761 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25027491339264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332736487 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1145032509260800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181688732 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1163070459478528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181711838 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181688732 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884364892893827e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚586 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚691 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1163070459478528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1145032509260800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1145032535652352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90294840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1145032535652352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1143566707127335 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1163073591263744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151065039 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156525203 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1145164029740544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1003527797407232 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1143695084024759 KW,àQ°EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1125685531703296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1145164003348992 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1163204065880576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚568 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1145164029740544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156525774 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1125685506275328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332774861 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332797999 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25030349489664 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _step‚692 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18009577749431 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 trainer/global_step‚8449 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181709822 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚978497388722176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18012681218563 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18040062531584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90305093 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚978497360587776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181709822 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156525203 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1145164029740544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1003527797407232 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1143695084024759 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1125685531703296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1145164003348992 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1163204065880576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚568 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1145164029740544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156525774 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1125685506275328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332774861 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332797999 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25030349489664 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _step‚692 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18009577749431 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 trainer/global_step‚8449 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181709822 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚978497388722176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18012681218563 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18040062531584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90305093 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚978497360587776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181709822 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151065039 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884365574142165e+09 _step‚694 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚25093094068224 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1151263490766848 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690840 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 151936434 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139612672 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚18095046612992 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 151936439 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27964 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚90911918 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚66581899 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚18095046612992 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 334320078 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884365574142165e+09 _step‚694 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚25093094068224 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1151263490766848 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690840 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 151936434 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139612672 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚18095046612992 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 151936439 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27964 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚90911918 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚66581899 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚18095046612992 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 334320078 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1151353083376640 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139612672 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 121964032 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1169448628006912 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚18095544630272 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 151949632 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚148 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1149680107940619 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 182373055 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18278 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884365584797351e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606656 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 151949637 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚66587174 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1008816140655104 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 151949632 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚146 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚18095544630272 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182391333 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚25093592085504 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 trainer/global_step‚8462 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚983722566218240 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 334340970 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1151353059082240 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚18064935208715 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚18062607517875 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1151353083376640 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚18093215023616 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1151353083376640 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139612672 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 121964032 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1169448628006912 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚18095544630272 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 151949632 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚148 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1149680107940619 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 182373055 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18278 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884365584797351e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2329606656 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 151949637 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚66587174 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1008816140655104·]WÝ‘#‚(:† ÛÀåÔûŸ¹µ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚122.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07óEä|#‚(:ñ ÛÀåÔÍÒö¶ network.recv‚ 2.879811e+06 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.290756206174745* disk./.usagePercent‚19.124720019920254 disk.dm-1.out‚ 320.46484375 disk.sda3.out‚ 320.46484375 disk.sda.in‚0.203125 proc.cpu.threads‚17 network.sent‚ 2.1200099e+07 disk.sda3.in‚0.203125 disk.drbd2265.out‚0( proc.memory.percent‚18.0189847946167$ disk./.usageGB‚284.9721488952637# disk.dm-0.out‚65876.11962890625 disk.sda1.out‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚65876.11962890625 disk.drbd2266.out‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 215.01171875# proc.memory.rssMB‚ 2952.23046875 disk.nvme0n1.in‚ 466.90234375 disk.sda2.out‚0 disk.rbd1.out‚ 4.4736328125" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚65876.11962890625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0$ memory_percent‚98.68767261505127 disk.dm-0.in‚ 466.90234375 disk.drbd1716.in‚0 disk.sda.out‚ 320.46484375HV¢2‘#- _wandbruntime‚8551  _runtime‚8551ߧ·E\e’#Û¤ :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08426772058010101 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.38945797085762024 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3229212164878845 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3831101059913635 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 jet/accuracy_4_of_7‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19650204479694366 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902769327163696 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4444444477558136 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 _step‚696 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99358731508255 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_4_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23118659853935242 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35500991344451904 particle/accuracy_1_of_7‚1 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5312835574150085 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702127754688263 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8191586136817932 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3086056113243103 *loss/val_total_loss‚42.162250772587825 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4876684844493866 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.004117758013308048 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.997897207736969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10249925404787064 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010621736757457256 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.2832888066768646 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.34532004594802856 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23842433094978333 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 trainer/global_step‚8462 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_3_of_3‚0.02966015413403511 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.38060998916625977 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40096351504325867 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4837643504142761 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5187987089157104 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23294149339199066 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2271825224161148 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3333282768726349 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013589886948466301 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32885265350341797 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920313954353333 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2961013913154602 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5362350940704346 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_6_of_7‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1876809000968933 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3318246603012085 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.35660916566848755 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.22906586527824402 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22781217098236084 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2098539173603058 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.328603595495224 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9365207552909851 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5137520432472229 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175439119338989 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 epoch‚6 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6968702673912048 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33888882398605347 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06764570623636246 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995478391647339 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5047057867050171 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35082006454467773 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4454023241996765 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06280788034200668 +jet/accuracy_2_of_3‚0.23156803846359253 particle/accuracy_4_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04810859262943268 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3591652810573578 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_3_of_6‚0.07692307978868484 _runtime‚8551.199910528 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.029585538432002068 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5971153974533081 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4154411852359772 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22855515778064728 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23111110925674438 jet/accuracy_6_of_6‚0 particle/accuracy_2_of_6‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2203085571527481 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.24924583733081818 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23301739990711212 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1876809000968933 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3824065923690796 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.33807510137557983 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28501683473587036 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.21875 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9989210367202759 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3699420094490051 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13763216137886047 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7884365728426225e+09 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.33393391966819763 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 particle/accuracy_2_of_2‚1 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15888509154319763 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40478840470314026 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34752947092056274 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28895998001098633 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20019158720970154 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.25791168212890625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27429938316345215 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014158995822072029 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.42306002974510193 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3006535470485687 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2038840800523758 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 jet/accuracy_5_of_7‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.42592594027519226 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6430763602256775 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.384464293718338 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41297647356987 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 'particle/f_score‚0.3610828220844269 particle/accuracy_6_of_6‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5313962697982788 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.349886953830719 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39354854822158813 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5418792366981506 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22861815989017487 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25816166400909424 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0161544568836689 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09717261791229248 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28104519844055176 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28571146726608276 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.4156283438205719 particle/accuracy_5_of_7‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38189101219177246 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.489583283662796 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917217493057251 +jet/accuracy_3_of_4‚0.06531188637018204 particle/accuracy_3_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5705710053443909 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3272988498210907 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_5‚0.4785906672477722 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.039850637316703796 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3255952298641205 *jet/accuracy_2_of_4‚0.3563823699951172 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.33957523107528687 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4862464964389801 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27503642439842224 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_3_of_7‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5252763628959656 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19319704174995422 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23735179007053375 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.33393391966819763 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3726557791233063 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3745613694190979 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22861236333847046 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4980159103870392 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26717934012413025 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.26458740234375 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9862231612205505 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.33560606837272644 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.47039473056793213 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.35817816853523254 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_2_of_3‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013025950640439987 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7007794380187988 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36076390743255615 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904184341430664 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3120119869709015 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3490821421146393 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9821364283561707 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1910751760005951 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_3‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028145741671323776 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_pur»cq-ity‚0.5625 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166626930237 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5897436141967773 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 jet/accuracy_2_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.33055248856544495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936234354972839 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2643846869468689 -jet/accuracy_4_of_5‚0.0061123911291360855 6validation_average_jet_accuracy‚0.3948655128479004 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10372369736433029 +jet/accuracy_3_of_5‚0.09622375667095184 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40473252534866333 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.35567885637283325 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5106843113899231 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43798449635505676 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5241070985794067 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +particle/specificity‚0.9472048878669739 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 5 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38112872838974 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2337568700313568 particle/accuracy_2_of_5‚1 jet/accuracy_4_of_6‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2684266269207001 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 'particle/accuracy‚0.843538224697113 particle/accuracy_3_of_6‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.07421384006738663 particle/accuracy_2_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19105228781700134 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020059578120708466 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 jet/accuracy_7_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03726958483457565 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.036294352263212204 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2670210599899292 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.17105263471603394 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44863614439964294 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 jet/accuracy_5_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5780017375946045 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.38957828283309937 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30644431710243225 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16447897255420685 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09567394852638245 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34824520349502563 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9890963435173035 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26996904611587524 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5579429864883423 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3199999928474426 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20344828069210052 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992708683013916 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913803935050964 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4166666567325592 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03488908335566521 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2618817985057831 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_4‚0.8926678895950317 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24691356718540192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26612168550491333 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4019606113433838 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23078429698944092 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.378375381231308 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004728957079350948 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34633585810661316 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3783016502857208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4015151858329773 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16225935518741608 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 particle/accuracy_7_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992677569389343 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_3‚0.8297277092933655 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929855465888977 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1547134518623352 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5672612190246582 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49293896555900574 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29254382848739624 +jet/accuracy_2_of_2‚0.10394366830587387 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17650370299816132 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.31331712007522583 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4098670482635498 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1<t’ìR“#ठ8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40473252534866333 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.35567885637283325 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5106843113899231 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43798449635505676 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5241070985794067 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +particle/specificity‚0.9472048878669739 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 5 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38112872838974 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2337568700313568 particle/accuracy_2_of_5‚1 jet/accuracy_4_of_6‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2684266269207001 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 'particle/accuracy‚0.843538224697113 particle/accuracy_3_of_6‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.07421384006738663 particle/accuracy_2_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19105228781700134 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020059578120708466 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 jet/accuracy_7_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03726958483457565 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.036294352263212204 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2670210599899292 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.17105263471603394 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44863614439964294 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 jet/accuracy_5_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5780017375946045 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.38957828283309937 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30644431710243225 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16447897255420685 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09567394852638245 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34824520349502563 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9890963435173035 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26996904611587524 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5579429864883423 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3199999928474426 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20344828069210052 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992708683013916 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913803935050964 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4166666567325592 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03488908335566521 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2618817985057831 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_4‚0.8926678895950317 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24691356718540192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26612168550491333 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4019606113433838 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23078429698944092 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.378375381231308 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004728957079350948 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34633585810661316 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3783016502857208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4015151858329773 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16225935518741608 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 particle/accuracy_7_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992677569389343 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_3‚0.8297277092933655 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929855465888977 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1547134518623352 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5672612190246582 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49293896555900574 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29254382848739624 +jet/accuracy_2_of_2‚0.10394366830587387 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17650370299816132 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.31331712007522583 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4098670482635498 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08426772058010101 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.38945797085762024 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3229212164878845 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3831101059913635 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 jet/accuracy_4_of_7‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19650204479694366 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902769327163696 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4444444477558136 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 _step‚696 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99358731508255 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_4_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23118659853935242 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35500991344451904 particle/accuracy_1_of_7‚1 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5312835574150085 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702127754688263 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8191586136817932 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3086056113243103 *loss/val_total_loss‚42.162250772587825 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4876684844493866 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.004117758013308048 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.997897207736969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10249925404787064 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010621736757457256 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.2832888066768646 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.34532004594802856 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23842433094978333 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 trainer/global_step‚8462 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_3_of_3‚0.02966015413403511 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.38060998916625977 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40096351504325867 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4837643504142761 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5187987089157104 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23294149339199066 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2271825224161148 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3333282768726349 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013589886948466301 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32885265350341797 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920313954353333 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2961013913154602 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5362350940704346 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_6_of_7‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1876809000968933 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3318246603012085 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.35660916566848755 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.22906586527824402 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22781217098236084 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2098539173603058 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.328603595495224 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9365207552909851 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5137520432472229 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175439119338989 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 epoch‚6 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6968702673912048 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33888882398605347 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06764570623636246 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995478391647339 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5047057867050171 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35082006454467773 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4454023241996765 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06280788034200668 +jet/accuracy_2_of_3‚0.23156803846359253 particle/accuracy_4_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04810859262943268 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3591652810573578 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_3_of_6‚0.07692307978868484 _runtime‚8551.199910528 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.029585538432002068 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5971153974533081 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4154411852359772 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22855515778064728 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23111110925674438 jet/accuracy_6_of_6‚0 particle/accuracy_2_of_6‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2203085571527481 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.24924583733081818 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23301739990711212 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.361³È´{?1111342906952 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1876809000968933 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3824065923690796 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.33807510137557983 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28501683473587036 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.21875 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9989210367202759 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3699420094490051 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13763216137886047 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7884365728426225e+09 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.33393391966819763 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 particle/accuracy_2_of_2‚1 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15888509154319763 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40478840470314026 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34752947092056274 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28895998001098633 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20019158720970154 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.25791168212890625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27429938316345215 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014158995822072029 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.42306002974510193 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3006535470485687 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2038840800523758 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 jet/accuracy_5_of_7‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.42592594027519226 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6430763602256775 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.384464293718338 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41297647356987 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 'particle/f_score‚0.3610828220844269 particle/accuracy_6_of_6‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5313962697982788 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.349886953830719 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39354854822158813 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5418792366981506 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22861815989017487 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25816166400909424 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0161544568836689 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09717261791229248 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28104519844055176 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28571146726608276 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.4156283438205719 particle/accuracy_5_of_7‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38189101219177246 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.489583283662796 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917217493057251 +jet/accuracy_3_of_4‚0.06531188637018204 particle/accuracy_3_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5705710053443909 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3272988498210907 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_5‚0.4785906672477722 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.039850637316703796 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3255952298641205 *jet/accuracy_2_of_4‚0.3563823699951172 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.33957523107528687 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4862464964389801 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27503642439842224 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_3_of_7‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5252763628959656 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19319704174995422 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23735179007053375 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.33393391966819763 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3726557791233063 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3745613694190979 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22861236333847046 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4980159103870392 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26717934012413025 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.26458740234375 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9862231612205505 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.33560606837272644 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.47039473056793213 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.35817816853523254 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_2_of_3‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013025950640439987 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7007794380187988 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36076390743255615 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9904184341430664 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3120119869709015 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3490821421146393 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9821364283561707 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1910751760005951 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_3‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028145741671323776 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166626930237 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5897436141967773 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 jet/accuracy_2_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.33055248856544495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936234354972839 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2643846869468689 -jet/accuracy_4_of_5‚0.0061123911291360855 6validation_average_jet_accuracy‚0.3948655128479004 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10372369736433029 +jet/accuracy_3_of_5‚0.09622375667095184 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625¸i#øžž”#‚(:“ êÀåÔ§ž§· gpu.0.gpu‚47.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚300.629' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.125800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó÷•}•#‚(:ò êÀåÔñ½ï¸) proc.memory.percent‚18.01908016204834 cpu‚25.290935071464002 network.recv‚ 2.884925e+06 disk.dm-0.in‚ 466.90234375 disk.drbd2265.out‚0' proc.memory.availableMB‚ 215.7109375 disk.sda1.out‚0 disk.drbd2265.in‚0* disk./.usagePercent‚19.125401255968868 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375# disk.dm-0.out‚65919.34619140625 disk.sda.out‚ 320.91015625 proc.cpu.threads‚17 network.sent‚ 2.1307496e+07$ disk./.usageGB‚284.9822998046875( disk.nvme0c0n1.out‚65919.34619140625 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 320.91015625 disk.rbd1.out‚ 4.53662109375 disk.sda3.out‚ 320.91015625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚65919.34619140625 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda2.in‚0 disk.drbd2266.out‚0$ memory_percent‚98.68340492248535# proc.memory.rssMB‚ 2952.24609375\çO9“–#‚(:ˆ ùÀåÔ½•Æµ gpu.0.gpu‚47.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚299.624' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f}³œ{—#‚(:ð ùÀåÔ–§ï¶ disk.rbd1.in‚0( disk.nvme0c0n1.out‚65960.91259765625 disk.nvme0n1.in‚ 466.90234375 disk.drbd2080.in‚0 disk.sda.out‚ 321.5078125 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚65960.91259765625 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.sda3.out‚ 321.5078125 disk.sda1.out‚0( proc.memory.availableMB‚ 215.46484375$ disk./.usageGB‚284.9810218811035 disk.sda2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.891524e+06* disk./.usagePercent‚19.125315493444898 disk.drbd2080.out‚0 disk.drbd2265.in‚0 cpu‚25.290556390197157 disk.sda3.in‚0.203125& disk.nvme0n1.out‚65960.91259765625 network.sent‚ 2.1431544e+07 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.90234375$ memory_percent‚98.68490695953369# proc.memory.rssMB‚ 2952.24609375 disk.sda2.out‚0 disk.dm-1.out‚ 321.5078125 disk.rbd1.out‚ 4.53662109375 disk.drbd1716.in‚0 disk.drbd1716.out‚0) proc.memory.percent‚18.01908016204834 disk.sda.in‚0.203125 disk.drbd2265.out‚0œØ2˜#- _wandbruntime‚8585  _runtime‚8585ÿ5rvÈ™# trainer/global_step‚8499 % _timestamp‚1.7884366072900648e+09 _runtime‚ 8585.64510255 _step‚697 ' lr-AdamW/pg1‚5.0224666934090066e-05 ' lr-AdamW/pg2‚5.0224666934090066e-05î¯i)Íš#Ç % _timestamp‚1.7884366072900648e+09 _runtime‚ 8585.64510255 _step‚697 ' lr-AdamW/pg1‚5.0224666934090066e-05 ' lr-AdamW/pg2‚5.0224666934090066e-05 trainer/global_step‚8499¹oë’2›#- _wandbruntime‚8585  _runtime‚8585pãÞM(œ#—P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18174972401152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66932639 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚8499 $ _timestamp‚1.788436607290786e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76857344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚988747942179328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91416312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155525128811668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18178077794816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1175426516386304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25207544754688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335976667 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1175426516386304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 335999779 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18147319232660 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18144215763736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1175429648171520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66932138 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152743429 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183256344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183233238 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1013955438211072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183256344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚988747970313728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440j1 øù #œP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8585.646043947 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18174972401152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76857344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚988747942179328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91416312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155525128811668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18178077794816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1175426516386304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25207544754688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_bat•è¢)ch_start/active.all.freed‚ 335976667 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1175426516386304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 335999779 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18147319232660 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18144215763736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1175429648171520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66932138 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152743429 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183256344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183233238 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1013955438211072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183256344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚988747970313728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18174972401152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66932639 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚8499 $ _timestamp‚1.788436607290786e+09 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1155653505709092 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1175560122788352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152760719 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1175563254586880 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1014070900195328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1175563254586880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152760719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183277460 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91426568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚988860381177856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336015041 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336038179 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1137507206580224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1137507181152256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1157383064464896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152760713 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25210470307840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18146324556836 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183254328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183254328 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76844032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25210519017472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158366215 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1157383038073344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183277460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91426565 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18180190121984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1014070823351296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48709632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183254328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183254328 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76844032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25210519017472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158366215 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1157383038073344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183277460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91426565 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18180190121984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1014070823351296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48709632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1155653505709092 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1175560122788352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152760719 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1175563254586880 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1014070900195328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1175563254586880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152760719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183277460 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91426568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚988860381177856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336015041 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336038179 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1137507206580224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1137507181152256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1157383064464896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152760713 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25210470307840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18146324556836 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153607629 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1143847634211840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184291090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1182106850902528 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1163826274786304 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153607635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1163826274786304 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184314196 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184291090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153607635 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1019724058061824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚994368513520128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25355491624448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18280602507776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18283707901440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184314196 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18280602507776 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25355544541696 trainer/global_step‚8549 % _timestamp‚1.7884366549935398e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182106850902528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67314770 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18249669009944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18280602507776 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25355544541696 trainer/global_step‚8549 % _timestamp‚1.7884366549935398e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182106850902528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67314770 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18249669009944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153607629 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1143847634211840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184291090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1182106850902528 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1163826274786304 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153607635¾)¹2°#- _wandbruntime‚8634  _runtime‚8634KM¸v2#±#¢N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1163957742482944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337960231 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184312180 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚994480896249856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337960231 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153624913 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18285820228608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67321659 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18254881272176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1162231937913200 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1162228809016100 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1182240457304576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1163957742482944 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337937093 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1143977031213056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1019839420316160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚8634.253253912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1163957768874496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153624913 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91939215 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153624919 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚501 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184335312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1019839339277824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182243589103104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337937093 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18282714821632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182243589103104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18285820228608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153624919 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67322160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25358495931904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1143977056641024 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1163957768874496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25358443027968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159260874 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456MÆé-'²#§N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18282714821632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182243589103104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18285820228608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153624919 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67322160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25358495931904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1143977056641024 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1163957768874496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25358443027968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159260874 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1163957742482944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337960231 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184312180 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚994480896249856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337960231 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153624913 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18285820228608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67321659 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18254881272176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1162231937913200 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1162228809016100 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1182240457304576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1163957742482944 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337937093 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1143977031213056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1019839420316160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 _runtime‚8634.253253912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1163957768874496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153624913 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91939215 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153624919 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚501 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184335312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1019839339277824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182243589103104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337937093¿]r M2³#- _wandbruntime‚8634  _runtime‚8634§Ôª5^´#Ø 0loss/FRt2/assignment_loss‚3.1655979320920746 /loss/FRt3/assignment_loss‚3.059611976146698 1loss/FBt3/assignment_loss‚0.10845963605424913 /loss/FBt4/detection_loss‚1.8857891526487138 trainer/global_step‚8549 /loss/FRt3/detection_loss‚2.4508461610359302 .loss/FRt4/detection_loss‚2.945911611829485 1loss/SRqqt1/detection_loss‚0.8266840733339389 1loss/SRqqt4/detection_loss‚2.0572291003896837 %loss/total_loss‚40.18592663353964 1loss/SRqqt3/assignment_loss‚1.188472162373364 2loss/SRqqt4/assignment_loss‚1.7934467710079032 /loss/FRt2/detection_loss‚2.6159148051820953 /loss/FBt3/detection_loss‚2.3928718119859695 /loss/FBt1/detection_loss‚0.6246835900736707 /loss/FBt2/detection_loss‚1.8160750203662448 epoch‚7 /loss/FRt4/assignment_loss‚3.332451765026365 2loss/SRqqt1/assignment_loss‚0.9641097509488463 2loss/SRqqt2/assignment_loss‚1.8863815631812964 /loss/FRt1/detection_loss‚1.3430554455315524 0loss/FRt1/assignment_loss‚1.3139219311150638 1loss/FBt1/assignment_loss‚0.02100613471367237 1loss/SRqqt3/detection_loss‚1.7086792911474522 1loss/FBt4/assignment_loss‚0.22383030424761172 1loss/SRqqt2/detection_loss‚2.0030465370569472 _runtime‚8634.254466441 0loss/FBt2/assignment_loss‚0.4578501060508036 % _timestamp‚1.7884366558994784e+09 _step‚704#ÍÖšcµ#Ý /loss/FBt3/detection_loss‚2.3928718119859695 /loss/FBt1/detection_loss‚0.6246835900736707 /loss/FBt2/detection_loss‚1.8160750203662448 epoch‚7 /loss/FRt4/assignment_loss‚3.332451765026365 2loss/SRqqt1/assignment_loss‚0.9641097509488463 2loss/SRqqt2/assignment_loss‚1.8863815631812964 /loss/FRt1/detection_loss‚1.3430554455315524 0loss/FRt1/assignment_loss‚1.3139219311150638 1loss/FBt1/assignment_loss‚0.02100613471367237 1loss/SRqqt3/detection_loss‚1.7086792911474522 1loss/FBt4/assignment_loss‚0.22383030424761172 1loss/SRqqt2/detection_loss‚2.0030465370569472 _runtime‚8634.254466441 0loss/FBt2/assignment_loss‚0.4578501060508036 % _timestamp‚1.7884366558994784e+09 _step‚704 0loss/FRt2/assignment_loss‚3.1655979320920746 /loss/FRt3/assignment_loss‚3.059611976146698 1loss/FBt3/assignment_loss‚0.10845963605424913 /loss/FBt4/detection_loss‚1.8857891526487138 trainer/global_step‚8549 /loss/FRt3/detection_loss‚2.4508461610359302 .loss/FRt4/detection_loss‚2.945911611829485 1loss/SRqqt1/detection_loss‚0.8266840733339389 1loss/SRqqt4/detection_loss‚2.0572291003896837 %loss/total_loss‚40.18592663353964 1loss/SRqqt3/assignment_loss‚1.188472162373364 2loss/SRqqt4/assignment_loss‚1.7934467710079032 /loss/FRt2/detection_loss‚2.6159148051820953ÀRqgž¶#‚(:“ µÁåԭ篵 gpu.0.gpu‚74.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.839200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[èȇz·#‚(:ï µÁåÔÝÏ’· disk.sda.in‚0.203125$ memory_percent‚98.68636131286621 cpu‚25.28886028582459 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 4.599609375" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125& disk.nvme0n1.out‚66133.44287109375) proc.memory.percent‚18.01936626434326 disk.sda.out‚ 323.65234375 disk.nvme0n1.in‚ 466.90234375# disk.dm-0.out‚66133.44287109375 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 323.65234375 disk.rbd0.in‚0 disk.dm-1.out‚ 323.65234375 disk.drbd2265.in‚0' proc.memory.availableMB‚ 215.2265625# proc.memory.rssMB‚ 2952.29296875 network.sent‚ 2.1558006e+07 network.recv‚ 2.910591e+06 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚66133.44287109375* disk./.usagePercent‚19.126371524523638$ disk./.usageGB‚284.9967575073242 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0ì “¸#‚(:ˆ ÄÁåÔó‡µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Uf¨p¹#‚(:å ÄÁåÔÙô½¶ disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125' proc.memory.availableMB‚ 215.2109375# proc.memory.rssMB‚ 2952.29296875 cpu‚25.288422787581325 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚66181.2626953125 disk.sda.out‚ 324.359375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0) disk./.usagePercent‚19.12780542272292 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.in‚0.203125$ memory_percent‚98.68645668029785 disk.rbd0.out‚0% disk.nvme0n1.out‚66181.2626953125 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd2265.in‚0 network.sent‚ 2.1666004e+07 network.recv‚ 2.916257e+06 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 324.359375 disk.rbd1.in‚0" disk.dm-0.out‚66181.2626953125) proc.memory.percent‚18.01936626434326# disk./.usageGB‚285.018123626709 disk.sda2.out‚0 disk.rbd1.out‚ 4.599609375 disk.sda3.out‚ 324.359375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 ¥E’º#‚(:‡ ÓÁåÔÞ©ý´ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚274.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03›ëªs»#‚(:è ÓÁåÔñ‡Ç· disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.599609375 disk.dm-0.in‚ 466.90234375 disk.drbd1716.in‚0 network.recv‚ 2.920856e+06 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 325.109375 disk.drbd2265.out‚0$ memory_percent‚98.68643283843994) proc.memory.percent‚18.01936626434326$ disk./.usageGB‚285.0354461669922 disk.drbd2080.in‚0( proc.memory.availableMB‚ 215.21484375 cpu‚25.288033821940754 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0* disk./.usagePercent‚19.128967952935902 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 325.109375' disk.nvme0c0n1.out‚66226.0791015625" disk.dm-0.out‚66226.0791015625 network.sent‚ 2.1672042e+07 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 325.109375% disk.nvme0n1.out‚66226.0791015625 disk.sda3.in‚0.203125 disk.sda1.in‚0# proc.memory.rssMB‚ 2952.29296875¦£52¼#- _wandbruntime‚8680  _runtime‚8680þÑ¥¥Ç½#Á & lr-AdamW/pg1‚4.770465092178057e-05 & lr-AdamW/pg2‚4.770465092178057e-05 trainer/global_step‚8599 % _timestamp‚1.7884367026437752e+09 _runtime‚8680.998825214 _step‚705•ëO̾#Æ & lr-AdamW/pg1‚4.770465092178057e-05 & lr-AdamW/pg2‚4.770465092178057e-05 trainer/global_step‚8599 % _timestamp‚1.7884367026437752e+09 _runtime‚8680.998825214 _step‚705ÁÒÔ¬H2¿#- _wandbruntime‚8680  _runtime‚8680Éqo³(À#˜P % _timestamp‚1.7884367026445172e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154471829 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339843883 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1170400979195904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18389338008064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚999989028592128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67694198 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1025492373039104 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18386232614400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154471835 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185372048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1168672606528792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚999989056726528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25503263395328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18389338008064 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1188790317203968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1170400979195904 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 trainer/global_step‚8599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1025492291987456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25503316312576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1150317509700608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67694706 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1188787185418752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18355122256152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92441615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92441612 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1188790317203968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160135810 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1168675735425684 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185348942 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878Uà¦ö> Á#P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1168675735425684 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185348942 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 % _timestamp‚1.7884367026445172e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154471829 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339843883 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1170400979195904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18389338008064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚999989028592128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67694198 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1025492373039104 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18386232614400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154471835 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185372048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VADeviceStatsMonitor.on_train_batch_start/rl«Yåequested_bytes.all.freed‚1168672606528792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚999989056726528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25503263395328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18389338008064 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1188790317203968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1170400979195904 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 trainer/global_step‚8599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1025492291987456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25503316312576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1150317509700608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67694706 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1188787185418752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18355122256152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92441615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92441612 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1188790317203968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160135810Âq2]Y2Â#- _wandbruntime‚8681  _runtime‚8681%¿+è*'Ã#¤N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1168804112323108 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170532446892544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18360334518384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚494 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339859145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚8599 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1150446881273856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339882283 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185393164 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1025607648682496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1150446906701824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884367035371196e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160153638 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154489119 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18391450335232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67701773 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185370032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170532473284096 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154489119 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18357231049252 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154489113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92451865 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1025607729720832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185393164 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18388344928256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _step‚707 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339882283 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1000101439456256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92451868 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18391450335232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1000101467590656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1188923923619328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25506262130176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160154132 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185370032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67702264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18388344928256 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1170532446892544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1188923923619328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154489113fö/'Ä#©N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92451868 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18391450335232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1000101467590656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1188923923619328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25506262130176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160154132 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185370032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67702264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18388344928256 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1170532446892544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1188923923619328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154489113 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1168804112323108 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170532446892544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18360334518384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚494 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339859145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚8599 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1150446881273856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339882283 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185393164 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1025607648682496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1150446906701824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884367035371196e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160153638 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154489119 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18391450335232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67701773 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185370032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170532473284096 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154489119 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18357231049252 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154489113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92451865 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1025607729720832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185393164 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18388344928256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _step‚707 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339882283 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1000101439456256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096Êó7C2Å#- _wandbruntime‚8681  _runtime‚8681³§V]Æ#× 0loss/FRt1/assignment_loss‚1.4231567117902968 2loss/SRqqt4/assignment_loss‚1.3009926399741418 /loss/FRt2/detection_loss‚2.4026416825211565 1loss/FBt2/assignment_loss‚0.27662294963374734 /loss/FRt1/detection_loss‚1.4532667581760694 0loss/SRqqt1/detection_loss‚0.771961322799325 1loss/SRqqt2/detection_loss‚2.3838919137205394 /loss/FBt1/detection_loss‚0.6314694141270593 /loss/FRt4/detection_loss‚3.0064498856663704 1loss/SRqqt3/detection_loss‚1.7633657266851515 /loss/FBt3/detection_loss‚2.4532782807946205 .loss/FBt4/detection_loss‚2.245603037731988 /loss/FBt3/assignment_loss‚0.143463322390744 2loss/SRqqt1/assignment_loss‚0.7872385803610087 1loss/SRqqt3/assignment_loss‚1.182984322142147 epoch‚7 trainer/global_step‚8599 % _timestamp‚1.7884367035390146e+09 _step‚708 0loss/FRt3/assignment_loss‚2.6631303214012307 /loss/FBt2/detection_loss‚1.9485418037934736 _runtime‚8681.893972028 /loss/FRt2/assignment_loss‚3.552469201709913 /loss/FRt3/detection_loss‚2.2660763606112053 1loss/SRqqt4/detection_loss‚2.2762493032675524 %loss/total_loss‚40.67970355709873 0loss/FRt4/assignment_loss‚3.7907612125078836 1loss/SRqqt2/assignment_loss‚1.773290025975023 2loss/FBt1/assignment_loss‚0.037093057910491556 1loss/FBt4/assignment_loss‚0.14570572140759655sIqbÇ#Ü /loss/FRt2/assignment_loss‚3.552469201709913 /loss/FRt3/detection_loss‚2.2660763606112053 1loss/SRqqt4/detection_loss‚2.2762493032675524 %loss/total_loss‚40.67970355709873 0loss/FRt4/assignment_loss‚3.7907612125078836 1loss/SRqqt2/assignment_loss‚1.773290025975023 2loss/FBt1/assignment_loss‚0.037093057910491556 1loss/FBt4/assignment_loss‚0.14570572140759655 0loss/FRt1/assignment_loss‚1.4231567117902968 2loss/SRqqt4/assignment_loss‚1.3009926399741418 /loss/FRt2/detection_loss‚2.4026416825211565 1loss/FBt2/assignment_loss‚0.27662294963374734 /loss/FRt1/detection_loss‚1.4532667581760694 0loss/SRqqt1/detection_loss‚0.771961322799325 1loss/SRqqt2/detection_loss‚2.3838919137205394 /loss/FBt1/detection_loss‚0.6314694141270593 /loss/FRt4/detection_loss‚3.0064498856663704 1loss/SRqqt3/detection_loss‚1.7633657266851515 /loss/FBt3/detection_loss‚2.4532782807946205 .loss/FBt4/detection_loss‚2.245603037731988 /loss/FBt3/assignment_loss‚0.143463322390744 2loss/SRqqt1/assignment_loss‚0.7872385803610087 1loss/SRqqt3/assignment_loss‚1.182984322142147 epoch‚7 trainer/global_step‚8599 % _timestamp‚1.7884367035390146e+09 _step‚708 0loss/FRt3/assignment_loss‚2.6631303214012307 /loss/FBt2/detection_loss‚1.9485418037934736 _runtime‚8681.893972028ÄR›¾žÈ#‚(:“ âÁåÔíÖÒµ gpu.0.gpu‚59.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚293.129' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.625800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.wyÉ#‚(:î âÁåÔµÝз disk.drbd2266.out‚0( disk.nvme0c0n1.out‚66278.30810546875 disk.drbd2265.out‚0) proc.memory.percent‚18.01936626434326 network.sent‚ 2.1677907e+07" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0& disk.nvme0n1.out‚66278.30810546875 disk.sda2.out‚0 disk.dm-1.out‚ 325.5390625' proc.memory.availableMB‚ 218.4296875 disk.rbd1.out‚ 4.599609375 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125 cpu‚25.287623425293447 network.recv‚ 2.924637e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0* disk./.usagePercent‚19.131397464435842% disk./.usageGB‚285.07164764404297 disk.sda3.out‚ 325.5390625 disk.drbd1716.out‚0 disk.sda.out‚ 325.5390625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚98.66681098937988 proc.cpu.threads‚17# disk.dm-0.out‚66278.30810546875 disk.nvme0n1.in‚ 466.90234375# proc.memory.rssMB‚ 2952.29296875ò¤6žÊ#‚(:“ ñÁåÔó‹µ gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚290.178' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.035599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áö§ãË#‚(:æ ñÁåÔžø¶& disk.nvme0c0n1.out‚66315.341796875 disk.rbd0.in‚0$ disk./.usageGB‚285.0946235656738$ disk.nvme0n1.out‚66315.341796875 disk.rbd1.out‚ 4.599609375! disk.dm-0.out‚66315.341796875 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125' proc.memory.availableMB‚ 218.1796875) disk./.usagePercent‚19.13293939781457 disk.rbd1.in‚0 disk.sda.out‚ 326.0859375 disk.rbd0.out‚0$ memory_percent‚98.66833686828613 disk.dm-1.in‚0.203125 disk.sda1.out‚0 network.sent‚ 2.1787118e+07 disk.dm-1.out‚ 326.0859375 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0) proc.memory.percent‚18.01936626434326 cpu‚25.287219088975775 disk.dm-0.in‚d ÔÔ 466.90234375 disk.sda3.out‚ 326.0859375 disk.sda1.in‚0# proc.memory.rssMB‚ 2952.29296875 network.recv‚ 2.931982e+06 disk.nvme0n1.in‚ 466.90234375 disk.sda2.in‚0 disk.drbd2266.out‚0À©S'“Ì#‚(:ˆ €ÂåÔÙµ gpu.0.gpu‚63.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûF’tÍ#‚(:é €ÂåÔÑïͶ disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚66364.2861328125* disk./.usagePercent‚19.135652821669986 disk.sda.out‚ 326.5859375 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2952.29296875" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 218.0234375 cpu‚25.286813995545323 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 326.5859375' disk.nvme0c0n1.out‚66364.2861328125 disk.sda1.out‚0$ memory_percent‚98.66929054260254% disk.nvme0n1.out‚66364.2861328125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.599609375 disk.sda3.out‚ 326.5859375) proc.memory.percent‚18.01936626434326 proc.cpu.threads‚17$ disk./.usageGB‚285.1350555419922 disk.rbd0.in‚0 network.sent‚ 2.179251e+07 network.recv‚ 2.935451e+06 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0ÆR2Î#- _wandbruntime‚8728  _runtime‚8728Cm!DÉÏ#à _runtime‚8728.532203901 _step‚709 ' lr-AdamW/pg1‚4.6461573963812155e-05 ' lr-AdamW/pg2‚4.6461573963812155e-05 trainer/global_step‚8649 % _timestamp‚1.7884367501771436e+09“ü•³ÎÐ#È ' lr-AdamW/pg1‚4.6461573963812155e-05 ' lr-AdamW/pg2‚4.6461573963812155e-05 trainer/global_step‚8649 % _timestamp‚1.7884367501771436e+09 _runtime‚8728.532203901 _step‚709Å^"('2Ñ#- _wandbruntime‚8728  _runtime‚8728õ€iˆ (Ò#šP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161027811 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155336029 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25650566637056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186429900 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1176975657213952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92954262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1156787359761408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 % _timestamp‚1.7884367501780207e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195467519934976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68073026 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚8649 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1176975683605504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18494968114688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186406794 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1031260166569984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1175247909835800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155336035 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18491862721024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341765935 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341742823 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1031260085518336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195470651720192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195470651720192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005609599932928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155336035 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186429900 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161027288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1176975657213952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _runtime‚8728.533573436 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25650513719808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18460575502360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186406794 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005609571798528 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195467519934976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68073026 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚8649 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1176975683605504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18494968114688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186406794 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1031260166569984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1175247909835800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155336035 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18491862721024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341765935 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341742823 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1031260085518336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195470651720192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195470651720192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005609599932928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155336035 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186429900 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161027288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1176975657213952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _runtime‚8728.533573436 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25650513719808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18460575502360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186406794 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005609571798528 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341804335 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341781197 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18493975034880 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1031375437457408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25653454794752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18462684295460 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18493975034880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341781197 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186451016 $ _timestamp‚1.788436751081979e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155353319 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186451016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155353313 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _runtime‚8729.437649785 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1177107151302144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155353313 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1177107177693696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18497080441856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161045772 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161045278 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68081254 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1175382544527216 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1195604258135552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186427884 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92964515 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1031375518495744 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1156916731334656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1195604258135552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1195601126337024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1177107151302144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.curreÓTcòÃnt‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68080763 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1005722010797056<ý8.'Ö#¨N EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25653454794752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18462684295460 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18493975034880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341781197 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186451016 $ _timestamp‚1.788436751081979e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155353319 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186451016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155353313 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 _runtime‚8729.437649785 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1177107151302144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155353313 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1177107177693696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18497080441856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161045772 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161045278 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68081254 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1175382544527216 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1195604258135552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186427884 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92964515 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1031375518495744 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1156916731334656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1195604258135552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1195601126337024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1177107151302144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68080763 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1005722010797056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25653507698688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1175379415630116 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18465787764592 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚8649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155353319 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341804335 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341781197 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18493975034880 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1031375437457408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132ÇfÇz 2×#- _wandbruntime‚8729  _runtime‚8729¡—Š_Ø#Ù 1loss/FBt4/assignment_loss‚0.14132564975362685 /loss/FRt1/detection_loss‚1.3030581366519134 1loss/SRqqt2/detection_loss‚2.2447817063913114 /loss/FBt2/detection_loss‚2.5305937138887553 _runtime‚8729.438911655 epoch‚7 2loss/SRqqt1/assignment_loss‚0.8591379347922546 2loss/SRqqt4/assignment_loss‚0.9420194886624813 .loss/FRt4/detection_loss‚2.588001969613527 2loss/SRqqt2/assignment_loss‚1.3351777582997228 /loss/FRt2/detection_loss‚2.4535698149885454 0loss/SRqqt3/detection_loss‚1.581655227026697 /loss/FBt4/detection_loss‚2.1454861561457315 %loss/total_loss‚39.68285142508979 trainer/global_step‚8649 $ _timestamp‚1.788436751083928e+09 0loss/FRt1/assignment_loss‚1.2823271862665813 /loss/FRt3/assignment_loss‚3.064281738322714 0loss/FRt4/assignment_loss‚3.0660726647627983 2loss/FBt1/assignment_loss‚0.000271787286715508 1loss/FBt2/assignment_loss‚0.10396059108181642 2loss/FBt3/assignment_loss‚0.019305368419736624 1loss/SRqqt3/assignment_loss‚1.516944543647943 /loss/FRt3/detection_loss‚2.4595448912485787 1loss/SRqqt1/detection_loss‚0.8402258682729943 1loss/SRqqt4/detection_loss‚2.2382273422554135 /loss/FBt3/detection_loss‚3.0693841576576233 /loss/FBt1/detection_loss‚0.7141539019278504 _step‚712 /loss/FRt2/assignment_loss‚3.183343827724457þ¶dÙ#Þ epoch‚7 2loss/SRqqt1/assignment_loss‚0.8591379347922546 2loss/SRqqt4/assignment_loss‚0.9420194886624813 .loss/FRt4/detection_loss‚2.588001969613527 2loss/SRqqt2/assignment_loss‚1.3351777582997228 /loss/FRt2/detection_loss‚2.4535698149885454 0loss/SRqqt3/detection_loss‚1.581655227026697 /loss/FBt4/detection_loss‚2.1454861561457315 %loss/total_loss‚39.68285142508979 trainer/global_step‚8649 $ _timestamp‚1.788436751083928e+09 0loss/FRt1/assignment_loss‚1.2823271862665813 /loss/FRt3/assignment_loss‚3.064281738322714 0loss/FRt4/assignment_loss‚3.0660726647627983 2loss/FBt1/assignment_loss‚0.000271787286715508 1loss/FBt2/assignment_loss‚0.10396059108181642 2loss/FBt3/assignment_loss‚0.019305368419736624 1loss/SRqqt3/assignment_loss‚1.516944543647943 /loss/FRt3/detection_loss‚2.4595448912485787 1loss/SRqqt1/detection_loss‚0.8402258682729943 1loss/SRqqt4/detection_loss‚2.2382273422554135 /loss/FBt3/detection_loss‚3.0693841576576233 /loss/FBt1/detection_loss‚0.7141539019278504 _step‚712 /loss/FRt2/assignment_loss‚3.183343827724457 1loss/FBt4/assignment_loss‚0.14132564975362685 /loss/FRt1/detection_loss‚1.3030581366519134 1loss/SRqqt2/detection_loss‚2.2447817063913114 /loss/FBt2/detection_loss‚2.5305937138887553 _runtime‚8729.438911655È̶8‹“Ú#‚(:ˆ ÂåÔÎä³µ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚261.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ͦ®kÛ#‚(:à ÂåÔž¢Ù· disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.sda2.out‚0) proc.memory.percent‚18.01936626434326 cpu‚25.286401779929705 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda.out‚327.3125 disk.drbd2265.out‚0 disk.rbd1.out‚ 4.599609375% disk.nvme0n1.out‚66399.6142578125 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375$ memory_percent‚98.66375923156738 proc.cpu.threads‚17) disk./.usagePercent‚19.13939565182058 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 466.90234375 network.sent‚ 2.1798541e+07 network.recv‚ 2.940096e+06$ disk./.usageGB‚285.1908264160156 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125' disk.nvme0c0n1.out‚66399.6142578125 disk.sda3.out‚327.3125' proc.memory.availableMB‚ 218.9296875 disk.dm-1.out‚327.3125 disk.rbd1.in‚0" disk.dm-0.out‚66399.6142578125# proc.memory.rssMB‚ 2952.29296875 disk.rbd0.out‚0)äsœÜ#‚(:‘ žÂåÔ΄þ¶ gpu.0.gpu‚80.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚284.22' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.84400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Å¡eÝ#‚(:Ú žÂåÔÄø˜¹$ disk./.usageGB‚285.2119903564453 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0$ proc.memory.availableMB‚218.9375) proc.memory.percent‚18.01936626434326 network.recv‚ 2.946134e+06 disk.dm-0.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚66444.279296875 disk.rbd1.in‚0* disk./.usagePercent‚19.140815981620545 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚327.9375# proc.memory.rssMB‚ 2952.29296875 cpu‚25.28598845121004 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚66444.279296875 disk.sda3.out‚327.9375 disk.sda.in‚0.203125 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 4.599609375 disk.sda.out‚327.9375 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375! disk.dm-0.out‚66444.279296875 disk.sda3.in‚0.203125 disk.drbd2265.out‚0$ memory_percent‚98.66371154785156 proc.cpu.threads‚17 network.sent‚ 2.1907244e+07ëL˜Þ#‚(:’ ­ÂåÔ¯…µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚279.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ß2sß#‚(:è ­ÂåÔǪï¶ disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 network.sent‚ 2.1913424e+07" disk.dm-0.out‚66500.1416015625 disk.sda1.out‚0 disk.drbd2265.in‚0) proc.memory.percent‚18.01936626434326 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 328.5390625 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125 disk.rbd0.out‚0 cpu‚25.285599470472082* disk./.usagePercent‚19.144758497647842% disk./.usageGB‚285.27073669433594 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0 disk.sda.out‚ 328.5390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 328.5390625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375' disk.nvme0c0n1.out‚66500.1416015625 disk.drbd1716.out‚0$ proc.memory.availableMB‚218.6875# proc.memory.rssMB‚ 2952.29296875 network.recv‚ 2.951219e+06% disk.nvme0n1.out‚66500.1416015625" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2080.out‚0$ memory_percent‚98.66523742675781’‰•½žà#‚(:“ ¼ÂåÔ¬âø¶ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.046' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.209199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜[qžvá#‚(:ë ¼ÂåÔŠ–ý¸* disk./.usagePercent‚19.146585367412182 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚66569.0517578125 disk.sda3.out‚ 328.8828125 disk.rbd0.out‚0) proc.memory.percent‚18.01936626434326 disk.dm-1.out‚ 328.8828125 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375% disk./.usageGB‚285.29795837402344 disk.drbd2266.in‚0" disk.dm-0.out‚66569.0517578125$ memory_percent‚98.66223335266113# proc.memory.rssMB‚ 2952.29296875 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 network.recv‚ 2.954626e+06 disk.sda2.out‚0 disk.sda1.in‚0% disk.nvme0n1.out‚66569.0517578125 cpu‚25.285185274750717 network.sent‚ 2.1918793e+07 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.sda.out‚ 328.8828125 disk.drbd2265.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 219.1796875ì"2â#- _wandbruntime‚8776  _runtime‚8776툮¨Èã# _runtime‚8776.056083526 _step‚713 ' lr-AdamW/pg1‚4.5230130568611274e-05 ' lr-AdamW/pg2‚4.5230130568611274e-05 trainer/global_step‚8699 $ _timestamp‚1.788436797701034e+09úb¥ÑÍä#Ç trainer/global_step‚8699 $ _timestamp‚1.788436797701034e+09 _runtime‚8776.056083526 _step‚713 ' lr-AdamW/pg1‚4.5230130568611274e-05 ' lr-AdamW/pg2‚4.5230130568611274e-05Éã~ 2å#- _wandbruntime‚8776  _runtime‚8776óªHÃ(æ#™P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1181826342039700 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68451986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18569132217492 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1183550388015104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25797996101120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187487752 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156200229 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1202147854451200 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1202150986236416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156200229 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18600598221312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68451476 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18600598221312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161918901 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1037028058188800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161918388 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1181823213142808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93466915 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180‡aì Z ç#žP G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1181826342039700 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68451986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18569132217492 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1183550388015104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25797996101120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187487752 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156200229 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1202147854451200 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1202150986236416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156200229 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18600598221312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68451476 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18600598221312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161918901 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1037028058188800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161918388 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1181823213142808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93466915 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180Ê•2u2è#- _wandbruntime‚8776  _runtime‚8776WgØ('é#¢N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037143508505600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187485736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1183681855711744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1183681855711744 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 $ _timestamp‚1.788436798591175e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156217519 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187508868 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156217519 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1011342554003456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1183681882103296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187485736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18568137541668 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1181957847834224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18571241010800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 343726387 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1037143425370112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1163386606823424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156217513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25800954502144 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343726387 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343703249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161936685 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161936183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚8699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1163386581395456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 343703249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1202281460853248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18599605141504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18602710548480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18599605141504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187508868 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93477168 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156217513 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68459517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68459018 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25800899501056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1202284592651776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93477165 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1011342525869056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 _runtime‚ 8776.94690628/“E?-'ê#§N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18602710548480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18599605141504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187508868 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93477168 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156217513 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68459517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68459018 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25800899501056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1202284592651776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93477165 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1011342525869056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 _runtime‚ 8776.94690628 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037143508505600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187485736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1183681855711744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1183681855711744 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 $ _timestamp‚1.788436798591175e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156217519 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187508868 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156217519 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1011342554003456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1183681882103296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187485736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18568137541668 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1181957847834224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18571241010800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 343726387 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1037143425370112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1163386606823424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156217513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25800954502144 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343726387 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343703249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161936685 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161936183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚8699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1163386581395456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 343703249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1202281460853248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18599605141504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976Ëá`«?2ë#- _wandbruntime‚8776  _runtime‚8776WD]x]ì#× 2loss/SRqqt4/assignment_loss‚1.7356587044229466 1loss/FBt1/assignment_loss‚0.01924791149735052 1loss/SRqqt1/detection_loss‚0.9865439666807652 1loss/SRqqt3/detection_loss‚1.6521812016444821 /loss/FBt4/detection_loss‚2.4051501899957657 %loss/total_loss‚40.04391851555005 % _timestamp‚1.7884367985930498e+09 /loss/FRt2/detection_loss‚2.4082873405479805 1loss/FBt2/assignment_loss‚0.11214029104707555 /loss/FRt1/detection_loss‚1.3790573894737228 epoch‚7 0loss/FRt1/assignment_loss‚1.5515563853325383 0loss/FRt2/assignment_loss‚3.1627879840571707 2loss/SRqqt3/assignment_loss‚1.2727242258767928 -loss/FRt4/detection_loss‚2.71814106667743 /loss/FBt1/detection_loss‚0.7304530025624177 0loss/FRt3/assignment_loss‚2.3522233976559206 2loss/SRqqt2/assignment_loss‚1.3923928610779144 1loss/FBt3/assignment_loss‚0.12172788129343341 1loss/SRqqt2/detection_loss‚2.0892519934757336 0loss/SRqqt4/detection_loss‚2.242917347805841 .loss/FBt2/detection_loss‚2.790227790673574 .loss/FRt3/detection_loss‚2.403579378669912 trainer/global_step‚8699 _runtime‚8776.948009512 _step‚716 0loss/FRt4/assignment_loss‚3.0869886559598587 1loss/SRqqt1/assignment_loss‚0.846105061173439 1loss/FBt4/assignment_loss‚0.17622080161527265 /loss/FBt3/detection_loss‚2.4083536863327026 •bí#Ü 0loss/FRt3/assignment_loss‚2.3522233976559206 2loss/SRqqt2/assignment_loss‚1.3923928610779144 1loss/FBt3/assignment_loss‚0.12172788129343341 1loss/SRqqt2/detection_loss‚2.0892519934757336 0loss/SRqqt4/detection_loss‚2.242917347805841 .loss/FBt2/detection_loss‚2.790227790673574 .loss/FRt3/detection_loss‚2.403579378669912 trainer/global_step‚8699 _runtime‚8776.948009512 _step‚716 0loss/FRt4/assignment_loss‚3.0869886559598587 1loss/SRqqt1/assignment_loss‚0.846105061173439 1loss/FBt4/assignment_loss‚0.17622080161527265 /loss/FBt3/detection_loss‚2.4083536863327026 2loss/SRqqt4/assignment_loss‚1.7356587044229466 1loss/FBt1/assignment_loss‚0.01924791149735052 1loss/SRqqt1/detection_loss‚0.9865439666807652 1loss/SRqqt3/detection_loss‚1.6521812016444821 /loss/FBt4/detection_loss‚2.4051501899957657 %loss/total_loss‚40.04391851555005 % _timestamp‚1.7884367985930498e+09 /loss/FRt2/detection_loss‚2.4082873405479805 1loss/FBt2/assignment_loss‚0.11214029104707555 /loss/FRt1/detection_loss‚1.3790573894737228 epoch‚7 0loss/FRt1/assignment_loss‚1.5515563853325383 0loss/FRt2/assignment_loss‚3.1627879840571707 2loss/SRqqt3/assignment_loss‚1.2727242258767928 -loss/FRt4/detection_loss‚2.71814106667743 /loss/FBt1/detection_loss‚0.7304530025624177Ì…Þ´“î#‚(:ˆ ËÂåÔÙ¦­µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚306.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.3418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è^Ÿ1vï#‚(:ë ËÂåÔ­­ƒ·* disk./.usagePercent‚19.150790547169613 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda2.in‚0' disk.nvme0c0n1.out‚66622.9228515625( proc.memory.availableMB‚ 218.41015625 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.dm-1.out‚ 329.5078125 disk.drbd1716.out‚0 network.sent‚ 2.1924864e+07 disk.rbd0.out‚0# proc.memory.rssMB‚ 2952.29296875 network.recv‚ 2.958983e+06 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 329.5078125 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.599609375$ memory_percent‚98.66693019866943) proc.memory.percent‚18.01936626434326 cpu‚25.284768421413922$ disk./.usageGB‚285.3606185913086 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚66622.9228515625 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.sda.out‚ 329.5078125 disk.drbd2265.out‚0% disk.nvme0n1.out‚66622.9228515625›±5äð#‚(:’ ÚÂåÔ¤‘ƒ¶ gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.12380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(2Ñ¢Ëñ#‚(:â ÚÂåÔ¦©ï¶' disk.nvme0c0n1.out‚66687.6396484375% disk./.usageGB‚285.39270782470703% disk.nvme0n1.out‚66687.6396484375 disk.rbd1.out‚ 4.599609375% proc.memory.availableMB‚!Akƒ¢ 218.65625 disk.sda2.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0# memory_percent‚98.6654281616211 network.recv‚ 2.965773e+06 disk.sda2.out‚0 disk.dm-0.in‚ 466.90234375* disk./.usagePercent‚19.152944082547645 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 329.96875 disk.drbd2266.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 329.96875 network.sent‚ 2.2033759e+07" disk.dm-0.out‚66687.6396484375 disk.sda3.in‚0.203125 disk.sda3.out‚ 329.96875 disk.sda.in‚0.203125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2952.29296875 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0 disk.drbd1716.out‚0) proc.memory.percent‚18.01936626434326 cpu‚25.284370618226156%W‚ª’ò#‚(:‡ éÂåÔî¨áµ gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚280.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.0462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ƒOnó#‚(:ã éÂåÔâìú· disk.drbd2080.out‚0 proc.cpu.threads‚17 network.sent‚ 2.2040159e+07 disk.dm-1.out‚ 330.609375 disk.rbd1.out‚ 4.599609375 disk.drbd2265.out‚0 cpu‚25.28396671181688* disk./.usagePercent‚19.155525150507856% disk./.usageGB‚285.43116760253906 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125' disk.nvme0c0n1.out‚66736.3623046875 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚66736.3623046875 disk.sda1.out‚0 disk.sda.out‚ 330.609375 disk.rbd0.out‚0$ memory_percent‚98.66561889648438 network.recv‚ 2.970587e+06 disk.nvme0n1.in‚ 466.90234375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0# proc.memory.availableMB‚218.625# proc.memory.rssMB‚ 2952.29296875) proc.memory.percent‚18.01936626434326" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 330.609375 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0% disk.nvme0n1.out‚66736.3623046875ôë`2ô#- _wandbruntime‚8823  _runtime‚8823®ê Éõ#à _runtime‚8823.484233818 _step‚717 ' lr-AdamW/pg1‚4.4010577367135676e-05 ' lr-AdamW/pg2‚4.4010577367135676e-05 trainer/global_step‚8749 % _timestamp‚1.7884368451292129e+09œ[ùÂÎö#È % _timestamp‚1.7884368451292129e+09 _runtime‚8823.484233818 _step‚717 ' lr-AdamW/pg1‚4.4010577367135676e-05 ' lr-AdamW/pg2‚4.4010577367135676e-05 trainer/global_step‚8749Í€or*2÷#- _wandbruntime‚8823  _runtime‚8823s†z‚ (ø#šP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1169727034455040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157064435 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1208831320752640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68830428 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1208831320752640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157064429 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188545604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18703122934272 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1042796058872832 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93979565 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25945372527104 % _timestamp‚1.7884368451300168e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042795975724032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188522498 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1190125066033152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1188401645346708 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157064435 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162809990 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157064429 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18674585463700 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345586927 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1190125066033152 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1016850658211328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1169727059883008 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18703122934272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25945317512704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1188398516449816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18671481994776 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1208828188967424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18706228327936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93979562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1208828188967424 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162810529 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18703122934272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25945317512704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1188398516449816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18671481994776 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1208828188967424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18706228327936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93979562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1208828188967424 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162810529 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1169727034455040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157064435 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1208831320752640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68830428 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1208831320752640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157064429 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188545604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18703122934272 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1042796058872832 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93979565 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25945372527104 % _timestamp‚1.7884368451300168e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042795975724032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188522498 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1190125066033152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1188401645346708 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157064435 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162809990 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157064429 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18674585463700 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345586927 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1190125066033152 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1016850658211328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1169727059883008 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968Î2¨+2ú#- _wandbruntime‚8824  _runtime‚8824;æd¹Ÿ#û#¤N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884368460240614e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1190256560121344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25948313163776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1188533151141232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345648439 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188543588 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18708340655104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18705235248128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188543588 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1016963069075456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1190256586512896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1188530022244132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚8749 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345648439 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157081713 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162827893 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93989818 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1042911410373632 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1169856456884224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1208964927168000 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1208961795369472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93989815 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1169856431456256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1208964927168000 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345625301 QBDeviceStatsMonitor.on_train_batch_end/active_bytesxC¶©‹.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157081719 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 _step‚719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18708340655104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1©aï+/'ü#©N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1190256560121344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25948313163776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1188533151141232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345648439 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188543588 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18708340655104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18705235248128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188543588 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1016963069075456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1190256586512896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1188530022244132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚8749 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345648439 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157081713 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162827893 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93989818 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1042911410373632 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1169856456884224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1208964927168000 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1208961795369472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93989815 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1169856431456256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1208964927168000 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345625301 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157081719 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 _step‚719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18708340655104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884368460240614e+09 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1215508523483648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26092177725952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18811858434560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _runtime‚8870.417818351 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189580350 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163699712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94492215 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1176196909943808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157928635 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18811858434560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18780038709908 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189603456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94492212 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1022471201417728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26092230643200 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1048563460195328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1215508523483648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1196699796834304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189580350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1194973819756824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1196699770442752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157928629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347532091 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69206977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1176196884515840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1215511655268864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1196699796834304]˜Óˆµ ‹$ŸP OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _runtime‚8870.417818351 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189580350 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163699712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94492215 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1176196909943808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157928635 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18811858434560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18780038709908 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189603456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94492212 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1022471201417728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26092230643200 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1048563460195328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pRJÆpool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1215508523483648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1196699796834304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189580350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1194973819756824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1196699770442752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157928629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347532091 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69206977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1176196884515840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1215511655268864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1196699796834304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1194976948653716 % _timestamp‚1.7884368920625377e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157928635 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347508979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157928629 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1215508523483648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26092177725952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18811858434560ÒÏŽPØ2Œ$- _wandbruntime‚8871  _runtime‚8871RÌ,Ô)'$£N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18813970761728 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1215645261684224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163717341 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157945919 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚ 8871.30922892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69214361 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1022583640416256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18782147503216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347570491 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347547353 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26095137027584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195108454448240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189601440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189601440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347570491 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157945919 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163716826 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1022583612281856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189624572 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157945913 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189624572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 % _timestamp‚1.7884368929535713e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚ 8871.30922892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69214361 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1022583640416256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18782147503216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347570491 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347547353 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26095137027584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195108454448240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189601440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189601440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347570491 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157945919 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163716826 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1022583612281856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189624572 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157945913 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189624572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 % _timestamp‚1.7884368929535713e+09 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1222188857999872 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1028091744624128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158792835 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158792829 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26239675874816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1222191989785088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95004865 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1028091772758528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164593367 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18885491956116 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1201552251960724 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚8917.688047161 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1203274474852352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚726 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1182666734576640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18917488541184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18917488541184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349454143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69587974 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190661308 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1203274501243904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190661308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69588502 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158792829 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349431031 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 trainer/global_step‚8849 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26239620860416 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1201549123063832 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1203274474852352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1201552251960724 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 _runtime‚8917.688047161 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1203274474852352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚726 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1182666734576640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18917488541184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18917488541184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349454143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69587974 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190661308 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1203274501243904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190661308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69588502 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158792829 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349431031 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 trainer/global_step‚8849 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26239620860416 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1201549123063832 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1203274474852352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1222188857999872 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1028091744624128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158792835 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158792829 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26239675874816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1222191989785088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95004865 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1028091772758528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164593367 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18885491956116 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884369402295136e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190659292 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8849 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1203405995332096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1028204155488256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95015118 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69596086 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26242569391104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164610693 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158810113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349492543 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190682424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69595578 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1054446808014848 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18884497280292 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158810119 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1203405995332096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1054446724879360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1201683757755248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚8918.585221028 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1028204183622656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1203405968940544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1182796131577856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1222325596200448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18919600868352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349469405 @:DeviceStatsMtÉ#2onitor.on_train_batch_end/num_sync_all_streams‚2(9Òå/' $©N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349469405 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18916495461376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26242624392192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349492543 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1182796157005824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190682424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1222325596200448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164611204 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _step‚727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190659292 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158810119 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349469405 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1222322464401920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884369402295136e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190659292 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8849 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1203405995332096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1028204155488256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95015118 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69596086 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26242569391104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164610693 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158810113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349492543 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190682424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69595578 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1054446808014848 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18884497280292 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158810119 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1203405995332096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1054446724879360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1201683757755248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚8918.585221028 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1028204183622656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1203405968940544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1182796131577856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1222325596200448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18919600868352×=ÿ{2¡$- _wandbruntime‚8918  _runtime‚8918 WVwZ¢$Ô /loss/FRt4/detection_loss‚3.2465386440356574 1loss/SRqqt4/detection_loss‚2.6105284183113664 /loss/FRt3/detection_loss‚2.5275441376786483 /loss/FBt1/detection_loss‚0.7012122618034482 .loss/FBt4/detection_loss‚2.133376517079093 epoch‚7 _step‚728 2loss/SRqqt3/assignment_loss‚1.4721078395421774 1loss/SRqqt2/detection_loss‚2.3452699303627016 _runtime‚8918.586312689 $ _timestamp‚1.788436940231358e+09 1loss/FBt4/assignment_loss‚0.28353803022764623 /loss/FRt4/assignment_loss‚2.980665221810341 1loss/SRqqt2/assignment_loss‚1.602845202681298 /loss/FRt1/detection_loss‚1.3967421777401265 -loss/FRt2/detection_loss‚2.35674643655156 trainer/global_step‚8849 0loss/FRt1/assignment_loss‚1.2889303317317715 /loss/FRt3/assignment_loss‚3.285742816172148 1loss/SRqqt4/assignment_loss‚1.291509653683062 2loss/FBt1/assignment_loss‚0.012252241635046346 0loss/SRqqt1/detection_loss‚0.944647911126199 1loss/SRqqt3/detection_loss‚1.6979389103516094 .loss/FBt2/detection_loss‚2.312650018930435 .loss/FBt3/detection_loss‚2.547078694616045 &loss/total_loss‚41.983577779594015 0loss/FRt2/assignment_loss‚3.7107985851376557 2loss/SRqqt1/assignment_loss‚0.8173840532038846 1loss/FBt2/assignment_loss‚0.24302713600918652 1loss/FBt3/assignment_loss‚0.17450260917290247#î%_£$Ù 1loss/SRqqt2/assignment_loss‚1.602845202681298 /loss/FRt1/detection_loss‚1.3967421777401265 -loss/FRt2/detection_loss‚2.35674643655156 trainer/global_step‚8849 0loss/FRt1/assignment_loss‚1.2889303317317715 /loss/FRt3/assignment_loss‚3.285742816172148 1loss/SRqqt4/assignment_loss‚1.291509653683062 2loss/FBt1/assignment_loss‚0.012252241635046346 0loss/SRqqt1/detection_loss‚0.944647911126199 1loss/SRqqt3/detection_loss‚1.6979389103516094 .loss/FBt2/detection_loss‚2.312650018930435 .loss/FBt3/detection_loss‚2.547078694616045 &loss/total_loss‚41.983577779594015 0loss/FRt2/assignment_loss‚3.7107985851376557 2loss/SRqqt1/assignment_loss‚0.8173840532038846 1loss/FBt2/assignment_loss‚0.24302713600918652 1loss/FBt3/assignment_loss‚0.17450260917290247 /loss/FRt4/detection_loss‚3.2465386440356574 1loss/SRqqt4/detection_loss‚2.6105284183113664 /loss/FRt3/detection_loss‚2.5275441376786483 /loss/FBt1/detection_loss‚0.7012122618034482 .loss/FBt4/detection_loss‚2.133376517079093 epoch‚7 _step‚728 2loss/SRqqt3/assignment_loss‚1.4721078395421774 1loss/SRqqt2/detection_loss‚2.3452699303627016 _runtime‚8918.586312689 $ _timestamp‚1.788436940231358e+09 1loss/FBt4/assignment_loss‚0.28353803022764623 /loss/FRt4/assignment_loss‚2.980665221810341Ø0îʤ$‚(:’ ÒÃåÔŸÚ³µ gpu.0.gpu‚64.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚300.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.08859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܴתw¥$‚(:ì ÒÃåÔ´Þü¶ disk.rbd0.out‚0 disk.drbd2266.in‚0 network.sent‚ 2.2285395e+07 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚0.203125) proc.memory.percent‚18.01936626434326 disk.drbd2080.in‚0 disk.sda.out‚ 334.62109375 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 466.90234375' proc.memory.availableMB‚ 218.4921875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚67163.9033203125 disk.sda3.out‚ 334.62109375$ memory_percent‚98.66642951965332 disk.sda.in‚0.203125% disk.nvme0n1.out‚67163.9033203125 disk.dm-1.out‚ 334.62109375 cpu‚25.28117195278455* disk./.usagePercent‚19.169424823571894$ disk./.usageGB‚285.6382827758789 disk.nvme0n1.in‚ 466.90234375" disk.dm-0.out‚67163.9033203125 network.recv‚ 3.003878e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 4.599609375" disk.nvme0c0n1.in‚ 466.90234375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2952.29296875ò(’¦$‚(:‡ áÃåԯⴠgpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚279.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.933 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°fâ•t§$‚(:é áÃåÔ÷®¨· disk.drbd1716.in‚0 cpu‚25.28078320152552 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 4.599609375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚98.66642951965332) proc.memory.percent‚18.01936626434326 disk.dm-0.in‚ 466.90234375 disk.sda.out‚ 335.1640625 disk.rbd0.in‚0 network.recv‚ 3.010955e+06* disk./.usagePercent‚19.170475734499952 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2952.29296875 network.sent‚ 2.2394388e+07 disk.dm-1.out‚ 335.1640625 disk.rbd1.in‚0" disk.dm-0.out‚67220.1572265625 disk.drbd2080.out‚0 disk.sda.in‚0.203125$ disk./.usageGB‚285.6539421081543' disk.nvme0c0n1.out‚67220.1572265625 disk.drbd2080.in‚0 disk.drbd2265.in‚0% disk.nvme0n1.out‚67220.1572265625' proc.memory.availableMB‚ 218.4921875 disk.sda3.out‚ 335.1640625 disk.rbd0.out‚0ÑÕ@¨$‚(:’ ðÃåÔ„‹¦µ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚297.386' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.47720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëK×t©$‚(:é ðÃåÔ—üä¶& disk.nvme0n1.out‚67281.95654296875 disk.rbd1.out‚ 4.599609375" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 335.515625 disk.sda1.in‚0 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 335.515625 network.sent‚ 2.2399719e+07 disk.sda2.out‚0 disk.drbd2266.out‚0# disk.dm-0.out‚67281.95654296875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 cpu‚25.280400932187593) disk./.usagePercent‚19.17193318539238 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2952.29296875) proc.memory.percent‚18.01936626434326$ disk./.usageGB‚285.6756591796875 disk.sda.out‚ 335.515625 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.in‚0$ memory_percent‚98.66642951965332 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚67281.95654296875 disk.sda3.in‚0.203125 disk.sda1.out‚0' proc.memory.availableMB‚ 218.4921875 proc.cpu.threads‚17 network.recv‚ 3.014624e+06 disk.drbd1716.out‚0ôMÇ~2ª$- _wandbruntime‚8965  _runtime‚8965ü„yÇ«$Á & lr-AdamW/pg1‚4.042578774889759e-05 & lr-AdamW/pg2‚4.042578774889759e-05 trainer/global_step‚8899 % _timestamp‚1.7884369868354583e+09 _runtime‚8965.190498889 _step‚729Dƃ̬$Æ % _timestamp‚1.7884369868354583e+09 _runtime‚8965.190498889 _step‚729 & lr-AdamW/pg1‚4.042578774889759e-05 & lr-AdamW/pg2‚4.042578774889759e-05 trainer/global_step‚8899ÙŒ­d2­$- _wandbruntime‚8965  _runtime‚8965W¨;(®$˜P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1209849205653504 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191719160 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1060099447240704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1208124426370840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159657035 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1208127555267732 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚730 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351376195 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159657029 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19020013254144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1189136584637440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18990945202324 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1228869192516096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1209849205653504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165489650 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1060099366189056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1228869192516096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351353083 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26387131275776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 # _timestamp‚1.78843698683616e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1228872324301312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1189136610065408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1228872324301312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165490176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26387078358528 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1209849179261952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚8965.191517542 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191696054 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1033712287830528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19020013254144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19023118647808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚8899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159657035 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1033712315964928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191696054 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964×á„ý¯$P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26387131275776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 # _timestamp‚1.78843698683616e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1228872324301312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1189136610065408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1228872324301312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165490176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26387078358528 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1209849179261952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚8965.191517542 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small»nR)&_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191696054 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1033712287830528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19020013254144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19023118647808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚8899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159657035 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1033712315964928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191696054 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1209849205653504 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191719160 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1060099447240704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1208124426370840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159657035 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1208127555267732 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚730 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351376195 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159657029 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19020013254144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1189136584637440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18990945202324 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1228869192516096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1209849205653504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165489650 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1060099366189056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1228869192516096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351353083Ú€¸þ´2°$- _wandbruntime‚8966  _runtime‚8966"’#*'±$¤N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191717144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351414595 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165507495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95527768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165507986 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159674313 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159674319 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1189265981638656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351391457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18989950526500 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26390045983232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _step‚731 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19025230974976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1033824698694656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚491 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26390098887168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18993053995632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060214825716224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1208259061062256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1209980699741696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159674313 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191717144 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159674319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060214744677888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191740276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69979730 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1209980673350144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456_ªHÓ/'²$©N `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1033824698694656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚491 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26390098887168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18993053995632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060214825716224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1208259061062256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1209980699741696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159674313 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191717144 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159674319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060214744677888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191740276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69979730 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1209980673350144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191717144 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351414595 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165507495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95527768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165507986 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159674313 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159674319 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1189265981638656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351391457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18989950526500 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26390045983232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 _step‚731 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19025230974976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ÛôÏgþ2³$- _wandbruntime‚8966  _runtime‚8966²·\=[´$Õ 1loss/FBt4/assignment_loss‚0.13569464514730498 /loss/FRt3/assignment_loss‚2.122602747332665 /loss/FRt4/assignment_loss‚2.862520746562792 1loss/FBt3/assignment_loss‚0.19260372803546488 /loss/FBt2/detection_loss‚2.2021297415097556 epoch‚7 trainer/global_step‚8899 &loss/total_loss‚40.510501896244534 % _timestamp‚1.7884369877364037e+09 /loss/FRt1/detection_loss‚1.9506715943238564 0loss/FRt2/assignment_loss‚3.2884042021390556 1loss/SRqqt2/assignment_loss‚1.340927640735907 2loss/SRqqt4/assignment_loss‚1.2918841941603298 /loss/SRqqt2/detection_loss‚2.05158000478619 /loss/FBt4/detection_loss‚2.2412060141563415 _step‚732 3loss/FBt1/assignment_loss‚0.0018546147015550646 1loss/FBt2/assignment_loss‚0.15375398807777352 .loss/FRt2/detection_loss‚2.555244854978613 /loss/SRqqt3/detection_loss‚1.58241111160286 1loss/SRqqt4/detection_loss‚2.3372366078968705 /loss/FBt1/detection_loss‚0.7305492263108696 .loss/FBt3/detection_loss‚2.882680594921112 2loss/SRqqt1/assignment_loss‚0.9658034393612883 .loss/FRt3/detection_loss‚2.814854941060466 1loss/SRqqt1/detection_loss‚1.0253741634323408 _runtime‚8966.091389375 0loss/FRt1/assignment_loss‚1.4686458259820938 1loss/SRqqt3/assignment_loss‚1.159161280870678 /loss/FRt4/detection_loss‚3.15270598815835030 ¿`µ$Ú 1loss/SRqqt3/assignment_loss‚1.159161280870678 /loss/FRt4/detection_loss‚3.1527059881583503 1loss/FBt4/assignment_loss‚0.13569464514730498 /loss/FRt3/assignment_loss‚2.122602747332665 /loss/FRt4/assignment_loss‚2.862520746562792 1loss/FBt3/assignment_loss‚0.19260372803546488 /loss/FBt2/detection_loss‚2.2021297415097556 epoch‚7 trainer/global_step‚8899 &loss/total_loss‚40.510501896244534 % _timestamp‚1.7884369877364037e+09 /loss/FRt1/detection_loss‚1.9506715943238564 0loss/FRt2/assignment_loss‚3.2884042021390556 1loss/SRqqt2/assignment_loss‚1.340927640735907 2loss/SRqqt4/assignment_loss‚1.2918841941603298 /loss/SRqqt2/detection_loss‚2.05158000478619 /loss/FBt4/detection_loss‚2.2412060141563415 _step‚732 3loss/FBt1/assignment_loss‚0.0018546147015550646 1loss/FBt2/assignment_loss‚0.15375398807777352 .loss/FRt2/detection_loss‚2.555244854978613 /loss/SRqqt3/detection_loss‚1.58241111160286 1loss/SRqqt4/detection_loss‚2.3372366078968705 /loss/FBt1/detection_loss‚0.7305492263108696 .loss/FBt3/detection_loss‚2.882680594921112 2loss/SRqqt1/assignment_loss‚0.9658034393612883 .loss/FRt3/detection_loss‚2.814854941060466 1loss/SRqqt1/detection_loss‚1.0253741634323408 _runtime‚8966.091389375 0loss/FRt1/assignment_loss‚1.4686458259820938Üe}B2“¶$‚(:ˆ ÿÃåÔ— gpu.0.gpu‚47.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Ö`¾x·$‚(:í ÿÃåÔ÷ØÍ· disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.01936626434326$ disk./.usageGB‚285.6936454772949 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚67340.28076171875 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2952.29296875' proc.memory.availableMB‚ 218.7109375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 336.1484375 disk.dm-0.in‚ 466.90234375$ memory_percent‚98.66509437561035 network.recv‚ 3.019093e+06* disk./.usagePercent‚19.173140260916323 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚67340.28076171875 proc.cpu.threads‚17 disk.sda3.out‚ 336.1484375 disk.sda.out‚ 336.1484375# disk.dm-0.out‚67340.28076171875 network.sent‚ 2.2405739e+07 disk.rbd1.out‚ 4.599609375 cpu‚25.280011940240513 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375Ñx„2¸$‚(:’ ŽÄåÔÀ¯µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.112' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.22240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RLl=w¹$‚(:ì ŽÄåÔöºÛ¶ network.recv‚ 3.025366e+06 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.90234375 disk.drbd2080.in‚0 disk.dm-1.out‚ 336.6015625$ memory_percent‚98.66361618041992 disk.rbd1.out‚ 4.599609375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125* disk./.usagePercent‚19.175472745560786 disk.sda3.in‚0.203125 disk.sda3.out‚ 336.6015625 disk.rbd0.in‚0% disk./.usageGB‚285.72840118408203 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚67399.70751953125 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.sent‚ 2.2514158e+07 disk.rbd1.in‚0# disk.dm-0.out‚67399.70751953125 disk.drbd1716.in‚0) proc.memory.percent‚18.01936626434326 cpu‚25.27960643959697 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 336.6015625 disk.drbd2265.in‚0& disk.nvme0n1.out‚67399.70751953125& proc.memory.availableMB‚ 218.953125# proc.memory.rssMB‚ 2952.29296875 proc.cpu.threads‚17øù–…“º$‚(:ˆ ÄåÔ¸ò¶µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚287.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡m}DU»$‚(:ã ÄåÔÏз disk.drbd2265.out‚0* disk./.usagePercent‚19.177449635740192& disk.nvme0n1.out‚67460.37548828125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚337.1875 disk.sda2.out‚0 disk.dm-1.out‚337.1875 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 \í’¡ disk.rbd0.out‚0) proc.memory.percent‚18.01936626434326$ disk./.usageGB‚285.7578582763672 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375 disk.drbd2265.in‚0$ memory_percent‚98.66514205932617& proc.memory.availableMB‚ 218.703125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚67460.37548828125# disk.dm-0.out‚67460.37548828125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda.in‚0.203125# proc.memory.rssMB‚ 2952.29296875 network.sent‚ 2.2520563e+07" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚337.1875 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.279222413269277 network.recv‚ 3.030264e+06 disk.nvme0n1.in‚ 466.90234375 disk.sda2.in‚00¨¹¨2¼$- _wandbruntime‚9012  _runtime‚9012}‹n‘ǽ$Á % _timestamp‚1.7884370342486117e+09 _runtime‚9012.603652056 _step‚733 & lr-AdamW/pg1‚3.925631128262929e-05 & lr-AdamW/pg2‚3.925631128262929e-05 trainer/global_step‚8949¨”À̾$Æ & lr-AdamW/pg1‚3.925631128262929e-05 & lr-AdamW/pg2‚3.925631128262929e-05 trainer/global_step‚8949 % _timestamp‚1.7884370342486117e+09 _runtime‚9012.603652056 _step‚733ÝDß â2¿$- _wandbruntime‚9012  _runtime‚9012Æw” (À$šP O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192777012 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1039332831036928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚9012.604722617 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1214699729677848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1216423883671552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19128748754432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19128748754432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26534099763200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1235549527032320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1216423910063104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160521229 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1195606434698240 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192777012 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1039332831036928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚9012.604722617 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1214699729677848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1216423883671552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19128748754432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19128748754432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26534099763200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1235549527032320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1216423910063104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160521229 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1195606434698240 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166399235 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19130861081600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1235686265232896 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19127755674624 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1039445270035456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚735 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166398728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353313509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1235686265232896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192774996 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192774996 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26536998697984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160538513 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1216555404151296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70358313 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1065982240599040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96040415 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160538519 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1214831235472164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19095403772708 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96040418 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1214834364369264 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353336647 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160538519 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192798128 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353336647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353313509 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1216555404151296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1216555377759744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2Ûw¾{ Ä$©N % _timestamp‚1.7884370351534505e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1235683133434368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1216555377759744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19098507241840 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1216555377759744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19127755674624 ]Dv;î&$JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19130861081600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70358817 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1195735857127424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192798128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166399235 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19130861081600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1235686265232896 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19127755674624 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1039445270035456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚735 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166398728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353313509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1235686265232896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192774996 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192774996 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26536998697984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160538513 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1216555404151296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70358313 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1065982240599040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96040415 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160538519 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1214831235472164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19095403772708 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96040418 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1214834364369264 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353336647 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160538519 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192798128 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353336647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353313509 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1216555404151296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1044953402377728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70732638 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355220299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1242232993333760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161385435 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1222998614472704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19234378861056 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167275453 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70732111 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167274923 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355197187 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1222998614472704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1071634748016128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1242229861548544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26681292721152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19231273467392 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193811758 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1221278161881748 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19198748225816 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1242232993333760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193834864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26681345638400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19231273467392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96542815 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1044953374243328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1202076310187008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19234378861056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96542812 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161385435 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161385429 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1221275032984856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1242229861548544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193811758 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1222998588081152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 % _timestamp‚1.7884370815431879e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1222998588081152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1044953402377728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70732638 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355220299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1242232993333760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161385435 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1222998614472704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19234378861056 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167275453 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70732111 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167274923 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355197187 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1222998614472704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1071634748016128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1242229861548544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26681292721152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19231273467392 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193811758 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1221278161881748 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19198748225816 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878âß(úS“Ô$‚(:ˆ ÙÄåÔ¢Úœ¶ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚132.727' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ú´-sÕ$‚(:è ÙÄåԳ' proc.memory.availableMB‚ 217.8515625 network.recv‚ 3.049586e+06 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda3.out‚ 339.3515625 disk.sda1.out‚0 disk.rbd1.out‚ 4.599609375( disk.nvme0c0n1.out‚67678.36669921875# disk.dm-0.out‚67678.36669921875 disk.drbd1716.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚67678.36669921875$ memory_percent‚98.67033958435059# proc.memory.rssMB‚ 2952.29296875' disk./.usagePercent‚19.184039013664 disk.drbd2080.out‚0 disk.rbd0.in‚0) proc.memory.percent‚18.01936626434326 cpu‚25.2777077661845$ disk./.usageGB‚285.8560447692871 disk.sda3.in‚0.203125 disk.sda.out‚ 339.3515625 disk.drbd2266.in‚0 network.sent‚ 2.2646807e+07 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 339.3515625 disk.rbd1.in‚0ú_›+2Ö$- _wandbruntime‚9060  _runtime‚9060†Q/ö*'×$¤N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1242366599749120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193832848 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355235561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167293162 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1071750020974080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9060.810439728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355258699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96553068 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19236491188224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355235561 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1202205681760256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19233385781248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161402713 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1223130082169344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355258699 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167292655 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161402713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1202205707188224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19203960488048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1242363467950592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193855980 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1045065813241856 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70740094 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19200857018916 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1223130108560896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19233385781248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193855980 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161402719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26684235866624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161402719g á/'Ø$©N PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167293162 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1071750020974080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9060.810439728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355258699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96553068 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19236491188224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355235561 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1202205681760256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19233385781248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161402713 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1223130082169344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355258699 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167292655 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161402713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1202205707188224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19203960488048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1242363467950592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193855980 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1045065813241856 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70740094 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19200857018916 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1223130108560896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19233385781248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193855980 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161402719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26684235866624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161402719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1242366599749120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193832848 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355235561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ãó2Ù$- _wandbruntime‚9060  _runtime‚9060²»±„^Ú$Ø % _timestamp‚1.7884370824565642e+09 2loss/SRqqt2/assignment_loss‚1.7370139135814764 1loss/SRqqt4/assignment_loss‚2.008331301808357 /loss/FRt4/detection_loss‚2.8405258339994095 /loss/FBt2/detection_loss‚1.8708016097545623 /loss/FRt1/detection_loss‚1.7363955911248923 .loss/FBt3/detection_loss‚2.656941346824169 .loss/FBt4/detection_loss‚2.123406220566143 _step‚740 0loss/FRt2/assignment_loss‚2.8591785694277565 0loss/FRt4/assignment_loss‚3.3402133478837857 2loss/FBt1/assignment_loss‚0.012591577130743591 1loss/FBt4/assignment_loss‚0.33641037076118996 1loss/SRqqt1/detection_loss‚0.9212635939756593 1loss/SRqqt2/detection_loss‚2.1539634115555706 %loss/total_loss‚40.97474578847892 /loss/FRt3/detection_loss‚2.4864943746258232 epoch‚7 0loss/FRt1/assignment_loss‚0.8421305623921481 2loss/SRqqt1/assignment_loss‚0.7923331512023112 0loss/FBt2/assignment_loss‚0.2665239302252296 .loss/FRt2/detection_loss‚2.292769520781761 0loss/SRqqt4/detection_loss‚2.323337419827779 trainer/global_step‚8999 _runtime‚9060.811563219 2loss/SRqqt3/assignment_loss‚1.5231129499152303 1loss/SRqqt3/detection_loss‚1.6860340908169746 0loss/FRt3/assignment_loss‚3.1998045023749855 1loss/FBt3/assignment_loss‚0.43618716888158815 /loss/FBt1/detection_loss‚0.5289814290413751ÿÃÀÙcÛ$Ý 2loss/SRqqt1/assignment_loss‚0.7923331512023112 0loss/FBt2/assignment_loss‚0.2665239302252296 .loss/FRt2/detection_loss‚2.292769520781761 0loss/SRqqt4/detection_loss‚2.323337419827779 trainer/global_step‚8999 _runtime‚9060.811563219 2loss/SRqqt3/assignment_loss‚1.5231129499152303 1loss/SRqqt3/detection_loss‚1.6860340908169746 0loss/FRt3/assignment_loss‚3.1998045023749855 1loss/FBt3/assignment_loss‚0.43618716888158815 /loss/FBt1/detection_loss‚0.5289814290413751 % _timestamp‚1.7884370824565642e+09 2loss/SRqqt2/assignment_loss‚1.7370139135814764 1loss/SRqqt4/assignment_loss‚2.008331301808357 /loss/FRt4/detection_loss‚2.8405258339994095 /loss/FBt2/detection_loss‚1.8708016097545623 /loss/FRt1/detection_loss‚1.7363955911248923 .loss/FBt3/detection_loss‚2.656941346824169 .loss/FBt4/detection_loss‚2.123406220566143 _step‚740 0loss/FRt2/assignment_loss‚2.8591785694277565 0loss/FRt4/assignment_loss‚3.3402133478837857 2loss/FBt1/assignment_loss‚0.012591577130743591 1loss/FBt4/assignment_loss‚0.33641037076118996 1loss/SRqqt1/detection_loss‚0.9212635939756593 1loss/SRqqt2/detection_loss‚2.1539634115555706 %loss/total_loss‚40.97474578847892 /loss/FRt3/detection_loss‚2.4864943746258232 epoch‚7 0loss/FRt1/assignment_loss‚0.8421305623921481ä,9xä“Ü$‚(:ˆ èÄåÔýΗµ gpu.0.gpu‚65.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚281.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅvìÊwÝ$‚(:ì èÄåÔ“™È·* disk./.usagePercent‚19.183860064397567 disk.sda3.in‚0.203125 disk.sda3.out‚ 339.87890625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0# memory_percent‚98.6689567565918 proc.cpu.threads‚17 network.sent‚ 2.2743308e+07 disk.drbd2266.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚67747.9189453125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 466.90234375 network.recv‚ 3.054953e+06 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 339.87890625' disk.nvme0c0n1.out‚67747.9189453125 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2952.29296875 cpu‚25.277330957450122" disk.nvme0c0n1.in‚ 466.90234375 disk.nvme0n1.in‚ 466.90234375% disk.nvme0n1.out‚67747.9189453125 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375 disk.rbd0.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 218.078125) proc.memory.percent‚18.01936626434326% disk./.usageGB‚285.85337829589844 disk.sda.out‚ 339.87890625¥É{éjÞ$‚(:ß ÷ÄåÔ™öò¶ disk.rbd0.in‚0% disk.nvme0n1.out‚67792.6142578125 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 466.90234375" disk.dm-0.out‚67792.6142578125 disk.sda.out‚340.625 cpu‚25.276988085785263 network.sent‚ 2.2841375e+07$ disk./.usageGB‚285.8722381591797 disk.drbd2265.out‚0 disk.rbd1.out‚ 4.685546875' disk.nvme0c0n1.out‚67792.6142578125 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0) proc.memory.percent‚18.01936626434326 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚340.625 disk.sda1.out‚0$ memory_percent‚98.67184162139893( proc.memory.availableMB‚ 217.60546875 network.recv‚ 3.061239e+06 disk.drbd1716.out‚0 disk.sda.in‚0.203125* disk./.usagePercent‚19.185125765646852 disk.drbd2265.in‚0 disk.dm-1.out‚340.625 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 466.90234375 disk.sda2.out‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2952.29296875Á‡û|’ß$‚(:‡ ÷ÄåÔÚ¤Ô· gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚130.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!©TDà$‚(:ˆ †ÅåԮﵵ gpu.0.gpu‚94.0 gpu.0.memory‚18+â”àiO gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚291.244' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z¼vá$‚(:ë †Ååԫ耸 network.recv‚ 3.065056e+06 disk.dm-1.out‚ 341.08984375 disk.sda3.in‚0.203125 disk.sda3.out‚ 341.08984375 disk.dm-0.in‚ 466.90234375 disk.sda.in‚0.203125# proc.memory.rssMB‚ 2952.29296875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 466.90234375 disk.drbd1716.in‚0% disk.nvme0n1.out‚67856.4638671875 disk.sda2.out‚0' disk.nvme0c0n1.out‚67856.4638671875 disk.drbd2265.out‚0$ disk./.usageGB‚285.8182792663574 disk.sda1.out‚0 disk.rbd0.out‚0) proc.memory.percent‚18.01936626434326 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 cpu‚25.27660885173787 disk.drbd2080.in‚0) disk./.usagePercent‚19.18150453907502 disk.rbd1.in‚0 disk.rbd1.out‚ 4.685546875" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.drbd1716.out‚0' proc.memory.availableMB‚ 217.8515625 network.sent‚ 2.2847347e+07 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0" disk.dm-0.out‚67856.4638671875 disk.drbd2080.out‚0 disk.sda.out‚ 341.08984375 disk.rbd0.in‚0$ memory_percent‚98.670339584350594tÀ2â$- _wandbruntime‚9107  _runtime‚9107™-†¬Çã$Á _runtime‚9107.821472599 _step‚741 & lr-AdamW/pg1‚3.695700471602652e-05 & lr-AdamW/pg2‚3.695700471602652e-05 trainer/global_step‚9049 % _timestamp‚1.7884371294663577e+09°kX(Ìä$Æ & lr-AdamW/pg1‚3.695700471602652e-05 & lr-AdamW/pg2‚3.695700471602652e-05 trainer/global_step‚9049 % _timestamp‚1.7884371294663577e+09 _runtime‚9107.821472599 _step‚741åÏÕÚ2å$- _wandbruntime‚9107  _runtime‚9107g&ÆÄ(æ$™P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194869610 $ _timestamp‚1.788437129467177e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168165940 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1077402167029760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162249629 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19340008967680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194892716 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1248913327849984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97055462 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1050573945584128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19304201472024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357119239 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1229573292490752 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26828221445632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1227853465188756 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1208546160247808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19307304940948 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194869610 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97055465 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1229573318882304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1208546134819840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162249635 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19340008967680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162249629 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1229573292490752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1077402081783808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19336903574016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚9107.822461415 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357142351 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1248910196064768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1050573917449728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1248913327849984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1229573318882304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71110478 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚9049 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1248910196064768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168166472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26828164334080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚00sžó$(ç$žP H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1229573318882304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71110478 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚9049 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1248910196064768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168166472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26828164334080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194869610 $ _timestamp‚1.788437129467177e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168165940 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1077402167029760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162249629 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19340008967680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194892716 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1248913327849984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97055462 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1050573945584128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19304201472024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357119239 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1229573292490752 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26828221445632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1227853465188756 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1208546160247808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19307304940948 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194869610 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97055465 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1229573318882304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1208546134819840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162249635 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19340008967680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162249629 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1229573292490752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1077402081783808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19336903574016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚9107.822461415 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357142351 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1248910196064768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1050573917449728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1248913327849984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529æýLrH2è$- _wandbruntime‚9108  _runtime‚9108rç "*'é$¤N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162266919 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162266913 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19342121294848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357157613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26831164211200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168183579 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19306310265124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1249043802466816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19342121294848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357180751 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19339015887872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208675531821056 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1077517520659456 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19309413734256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357180751 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162266919 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1050686356448256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1227981842086180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚743 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚9108.726893273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97065718 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26831105015808 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1229704812970496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208675557249024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1249046934265344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194913832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1050686328313856Mº#Xê$©N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YFDŽTz×&eviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19339015887872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208675531821056 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1077517520659456 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19309413734256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357180751 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162266919 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1050686356448256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1227981842086180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚743 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚9108.726893273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97065718 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26831105015808 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1229704812970496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208675557249024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1249046934265344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194913832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1050686328313856 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162266919 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162266913 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19342121294848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357157613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26831164211200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168183579 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19306310265124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1249043802466816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19342121294848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357180751ç‰;ë2ë$- _wandbruntime‚9108  _runtime‚9108\àîa]ì$× epoch‚7 _step‚744 1loss/FBt2/assignment_loss‚0.01177321292925626 1loss/SRqqt1/detection_loss‚0.8661147791986601 /loss/FBt2/detection_loss‚2.9294407814741135 %loss/total_loss‚41.04977065524099 2loss/SRqqt1/assignment_loss‚0.5694174812517211 .loss/FBt3/detection_loss‚2.629098049231938 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚3.6874813391612125 2loss/SRqqt2/assignment_loss‚1.7464947257848347 2loss/FBt1/assignment_loss‚0.022543109975560267 /loss/FRt1/detection_loss‚1.3727018345324766 ,loss/FRt3/detection_loss‚2.4670290350914 0loss/SRqqt2/detection_loss‚2.288827575304929 /loss/FBt4/detection_loss‚2.0138164030181036 0loss/FRt4/assignment_loss‚3.5577502250671387 2loss/SRqqt3/assignment_loss‚1.0927237143175443 2loss/SRqqt4/assignment_loss‚1.4720999454813344 /loss/FRt4/detection_loss‚2.9050343963834973 /loss/FBt1/detection_loss‚0.7494538165887611 .loss/FRt2/detection_loss‚2.275387164873956 /loss/SRqqt3/detection_loss‚1.91161831215024 1loss/SRqqt4/detection_loss‚2.4308893297399794 0loss/FRt3/assignment_loss‚2.8204271346330643 2loss/FBt4/assignment_loss‚0.050553600581931986 % _timestamp‚1.7884371303731046e+09 _runtime‚9108.728080314 0loss/FRt1/assignment_loss‚1.0441786788758778 1loss/FBt3/assignment_loss‚0.13491600959345565ß:bí$Ü 0loss/FRt1/assignment_loss‚1.0441786788758778 1loss/FBt3/assignment_loss‚0.13491600959345565 epoch‚7 _step‚744 1loss/FBt2/assignment_loss‚0.01177321292925626 1loss/SRqqt1/detection_loss‚0.8661147791986601 /loss/FBt2/detection_loss‚2.9294407814741135 %loss/total_loss‚41.04977065524099 2loss/SRqqt1/assignment_loss‚0.5694174812517211 .loss/FBt3/detection_loss‚2.629098049231938 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚3.6874813391612125 2loss/SRqqt2/assignment_loss‚1.7464947257848347 2loss/FBt1/assignment_loss‚0.022543109975560267 /loss/FRt1/detection_loss‚1.3727018345324766 ,loss/FRt3/detection_loss‚2.4670290350914 0loss/SRqqt2/detection_loss‚2.288827575304929 /loss/FBt4/detection_loss‚2.0138164030181036 0loss/FRt4/assignment_loss‚3.5577502250671387 2loss/SRqqt3/assignment_loss‚1.0927237143175443 2loss/SRqqt4/assignment_loss‚1.4720999454813344 /loss/FRt4/detection_loss‚2.9050343963834973 /loss/FBt1/detection_loss‚0.7494538165887611 .loss/FRt2/detection_loss‚2.275387164873956 /loss/SRqqt3/detection_loss‚1.91161831215024 1loss/SRqqt4/detection_loss‚2.4308893297399794 0loss/FRt3/assignment_loss‚2.8204271346330643 2loss/FBt4/assignment_loss‚0.050553600581931986 % _timestamp‚1.7884371303731046e+09 _runtime‚9108.728080314èÃØî$‚(:„ •ÅåԈĵ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.6' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.52 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÛbuï$‚(:ê •ÅåÔèšÁ· network.sent‚ 2.2853639e+07 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 cpu‚25.276240043668604 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.out‚ 341.7421875 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚67918.5224609375" disk.dm-0.out‚67918.5224609375 disk.drbd2080.in‚0* disk./.usagePercent‚19.184245611744135 disk.nvme0n1.in‚ 466.90234375% disk.nvme0n1.out‚67918.5224609375 disk.sda2.in‚0 disk.sda3.out‚ 341.7421875 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0$ memory_percent‚98.67238998413086& proc.memory.availableMB‚ 217.515625# proc.memory.rssMB‚ 2952.29296875" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125) proc.memory.percent‚18.01936626434326 network.recv‚ 3.069821e+06% disk./.usageGB‚285.85912322998047 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 341.7421875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.748046875 disk.drbd1716.out‚0 proc.cpu.threads‚17¬vr“ð$‚(:ˆ ¤ÅåÔý¿ µ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚283.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯®}ñ$‚(:ò ¤ÅåÔÆðŒ· disk.sda.out‚ 342.26953125 disk.drbd2265.in‚0 disk.dm-1.out‚ 342.26953125 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚67992.84423828125# proc.memory.rssMB‚ 2952.29296875 disk.drbd2080.in‚0 disk.rbd1.in‚0) proc.memory.percent‚18.01936626434326 proc.cpu.threads‚17 network.recv‚ 3.075741e+06 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125# disk.dm-0.out‚67992.84423828125$ memory_percent‚98.67098331451416 network.sent‚ 2.2961777e+07* disk./.usagePercent‚19.186203557366007 disk.drbd2265.out‚0& disk.nvme0n1.out‚67992.84423828125 disk.sda3.in‚0.203125% disk./.usageGB‚285.88829803466797 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd1.out‚ 4.748046875 disk.sda3.out‚ 342.26953125( proc.memory.availableMB‚ 217.74609375 disk.sda2.out‚0 cpu‚25.275844588106164ÂV1è“ò$‚(:ˆ ³ÅåÔ†‚”µ gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚306.064' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.2128 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&øÆÂwó$‚(:ì ³ÅåÔùä¶ disk.sda1.in‚0 disk.rbd1.out‚ 4.7802734375 disk.sda2.out‚0 disk.dm-1.out‚ 342.8359375" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 342.8359375$ memory_percent‚98.67098331451416 network.recv‚ 3.080046e+06 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 217.74609375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0* disk./.usagePercent‚19.187631055376926$ disk./.usageGB‚285.9095687866211 disk.sda2.in‚0# disk.dm-0.out‚68049.34423828125 disk.sda3.out‚ 342.8359375 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.01936626434326 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2952.29296875 cpu‚25.27547873597682 proc.cpu.threads‚17 disk.nvme0n1.in‚ 466.90234375& disk.nvme0n1.out‚68049.34423828125 disk.drbd2266.in‚0 network.sent‚ 2.29678e+07( disk.nvme0c0n1.out‚68049.34423828125aBm2ô$- _wandbruntime‚9155  _runtime‚91553׿›Æõ$À $ _timestamp‚1.788437177230749e+09 _runtime‚9155.585764093 _step‚745 & lr-AdamW/pg1‚3.582765378773422e-05 & lr-AdamW/pg2‚3.582765378773422e-05 trainer/global_step‚9099ÏñžØËö$Å _runtime‚9155.585764093 _step‚745 & lr-AdamW/pg1‚3.582765378773422e-05 & lr-AdamW/pg2‚3.582765378773422e-05 trainer/global_step‚9099 $ _timestamp‚1.788437177230749e+09é5ôS2÷$- _wandbruntime‚9155  _runtime‚9155Õ#\< (ø$šP GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26974611889664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1083169015434240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71486632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1215015984880640 % _timestamp‚1.7884371772314296e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19442533680640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 359041291 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1255590530580992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195927462 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 359064403 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97568112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163113835 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1255590530580992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9155.586681756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1236147996900352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169054213 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19409654718232 _step‚746 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195950568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195927462 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19445639074304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1255593662366208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1234428768495764 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1236148023291904 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163113829 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚9099 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ô}Õ¤•ù$ŸP VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1215016010308608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1056194460656128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71486101 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1236148023291904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1234425639598872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163113829 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1083169015434240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71486632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1215015984880640 % _timestamp‚1.7884371772314296e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19442533680640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 359041291 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1255590530580992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195927462 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 359064403 TGDeviceStatsMonitor.on_train_batch_Ðôúkstart/inactive_split.large_pool.freed‚97568112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163113835 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1255590530580992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9155.586681756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1236147996900352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169054213 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19409654718232 _step‚746 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195950568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195927462 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19445639074304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1255593662366208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1234428768495764 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1236148023291904 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163113829 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚9099 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26974611889664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359102803 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1056306899654656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163131119 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚9099 _runtime‚9156.490092868 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163131113 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1236279517380096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163131113 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1236279490988544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1215145381881856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195971684 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169071855 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1234560274290288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1255724136983040 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1255727268781568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1083284449790976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19447751401472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97578368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71494002 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26977495135232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163131119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26977550136320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1215145407309824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1234557145393188 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19444645994496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19447751401472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0^Þ'.'ü$¨N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19447751401472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97578368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71494002 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26977495135232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163131119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26977550136320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1215145407309824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1234557145393188 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19444645994496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19447751401472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 359079665 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195948552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1255727268781568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1083284366655488 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056306871520256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359102803 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1056306899654656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163131119 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 trainer/global_step‚9099 _runtime‚9156.490092868 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163131113 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1236279517380096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163131113 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1236279490988544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1215145381881856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195971684 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169071855 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1234560274290288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1255724136983040 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1255727268781568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1083284449790976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ër2ý$- _wandbruntime‚9156  _runtime‚9156’K^þ$Ø .loss/FRt4/detection_loss‚3.097509140968323 2loss/FBt4/assignment_loss‚0.012559502721867628 1loss/SRqqt3/detection_loss‚1.9741455773738297 %loss/total_loss‚40.06480229957159 trainer/global_step‚9099 % _timestamp‚1.7884371781362336e+09 /loss/FRt4/assignment_loss‚3.153312611579895 /loss/FRt1/detection_loss‚1.7524408842264503 1loss/SRqqt3/assignment_loss‚1.208928396713666 2loss/SRqqt4/assignment_loss‚1.7581089337666829 .loss/FRt3/detection_loss‚2.400960391218012 1loss/SRqqt1/detection_loss‚0.8834218699061736 0loss/FRt1/assignment_loss‚1.4457952501597229 3loss/FBt1/assignment_loss‚0.0037398024707848486 .loss/FBt1/detection_loss‚0.559677171410017 /loss/FBt3/detection_loss‚1.7833872058174827 1loss/SRqqt4/detection_loss‚2.2928859121871716 /loss/FBt2/detection_loss‚2.1076425115267434 /loss/FBt4/detection_loss‚2.5860217412312827 epoch‚7 _runtime‚9156.491214045 0loss/FRt2/assignment_loss‚2.4980062110083443 /loss/FRt3/assignment_loss‚2.945682154460387 0loss/FBt3/assignment_loss‚0.2849051921670748 /loss/FRt2/detection_loss‚2.8124773349080767 0loss/SRqqt2/detection_loss‚2.179810546595475 _step‚748 2loss/SRqqt1/assignment_loss‚0.9511940670728852 2loss/SRqqt2/assignment_loss‚1.1913035420443991 1loss/FBt2/assignment_loss‚0.18088634803684223Êp‡cÿ$Ý 1loss/SRqqt3/assignment_loss‚1.208928396713666 2loss/SRqqt4/assignment_loss‚1.7581089337666829 .loss/FRt3/detection_loss‚2.400960391218012 1loss/SRqqt1/detection_loss‚0.8834218699061736 0loss/FRt1/assignment_loss‚1.4457952501597229 3loss/FBt1/assignment_loss‚0.0037398024707848486 .loss/FBt1/detection_loss‚0.559677171410017 /loss/FBt3/detection_loss‚1.7833872058174827 1loss/SRqqt4/detection_loss‚2.2928859121871716 /loss/FBt2/detection_loss‚2.1076425115267434 /loss/FBt4/detection_loss‚2.5860217412312827 epoch‚7 _runtime‚9156.491214045 0loss/FRt2/assignment_loss‚2.4980062110083443 /loss/FRt3/assignment_loss‚2.945682154460387 0loss/FBt3/assignment_loss‚0.2849051921670748 /loss/FRt2/detection_loss‚2.8124773349080767 0loss/SRqqt2/detection_loss‚2.179810546595475 _step‚748 2loss/SRqqt1/assignment_loss‚0.9511940670728852 2loss/SRqqt2/assignment_loss‚1.1913035420443991 1loss/FBt2/assignment_loss‚0.18088634803684223 .loss/FRt4/detection_loss‚3.097509140968323 2loss/FBt4/assignment_loss‚0.012559502721867628 1loss/SRqqt3/detection_loss‚1.9741455773738297 %loss/total_loss‚40.06480229957159 trainer/global_step‚9099 % _timestamp‚1.7884371781362336e+09 /loss/FRt4/assignment_loss‚3.153312611579895 /loss/FRt1/detection_loss‚1.7524408842264503ìÚ¾`ñž€%‚(:“ ÂÅåÔÛ¾©µ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚288.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.657199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aXtr%‚(:ç ÂÅåÔ¹’ã· disk.rbd1.out‚ 4.7802734375# disk.dm-0.out‚68111.60791015625 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0) proc.memory.percent‚18.01936626434326 disk.nvme0n1.in‚ 466.90234375 disk.sda.in‚0.203125* disk./.usagePercent‚19.190135833091933 disk.drbd2265.out‚0 disk.drbd2266.in‚0' proc.memory.availableMB‚ 217.4609375 disk.sda2.out‚0( disk.nvme0c0n1.out‚68111.60791015625 disk.sda3.in‚0.203125& disk.nvme0n1.out‚68111.60791015625 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 343.28125 network.sent‚ 2.297368e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 343.28125# memory_percent‚98.6727237701416# proc.memory.rssMB‚ 2952.29296875 cpu‚25.275130724384688 proc.cpu.threads‚17 network.recv‚ 3.084404e+06% disk./.usageGB‚285.94689178466797 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 343.28125/‘$‚%‚(:’ ÑÅåÔ¨í¥µ gpu.0.gpu‚65.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚288.667' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É/Õ~ƒ%‚(:ó ÑÅåÔ™íú¶ cpu‚25.274741424353422 disk.nvme0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 343.84765625 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚68164.54931640625# disk.dm-0.out‚68164.54931640625 network.recv‚ 3.091768e+06 disk.sda.out‚ 343.84765625 disk.sda2.in‚0 disk.rbd1.out‚ 4.84326171875$ memory_percent‚98.67424964904785% disk./.usageGB‚285.96741485595703 disk.rbd0.in‚0' proc.memory.availableMB‚ 217.2109375) proc.memory.percent‚18.01936626434326 network.sent‚ 2.3082708e+07 disk.dm-1.in‚0.203125 disk.sda3.out‚ 343.84765625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2952.29296875 disk.sda3.in‚0.203125* disk./.usagePercent‚19.191513153626147 disk.drbd2265.out‚0& disk.nvme0n1.out‚68164.54931640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 466.90234375R!B„%‚(:’ àÅåÔÔ‰µ gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚295.895' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.178999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û½5ÀÑ…%‚(:è àÅåÔÎŽ· disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.90234375 disk.rbd1.in‚0& proc.memory.availableMB‚ 217.203125 network.recv‚ 3.094951e+06) disk./.usagePercent‚19.19281495193776 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 466.90234375 network.sent‚ 2.3087999e+07% disk./.usageGB‚285.98681259155273 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0gãiÞ¢ disk.rbd1.out‚ 4.84326171875 disk.sda3.in‚0.203125" disk.dm-0.out‚68234.9638671875 disk.sda.out‚ 344.328125 disk.sda2.out‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚68234.9638671875 disk.sda3.out‚ 344.328125 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.percent‚18.01936626434326 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125$ memory_percent‚98.67429733276367# proc.memory.rssMB‚ 2952.29296875 cpu‚25.274347831496787 disk.rbd0.in‚0% disk.nvme0n1.out‚68234.9638671875 disk.dm-1.out‚ 344.328125 disk.drbd2266.in‚0 disk.sda1.out‚0²·ê:2†%- _wandbruntime‚9203  _runtime‚9203vG¬É‡%à _step‚749 ' lr-AdamW/pg1‚3.4712152515235646e-05 ' lr-AdamW/pg2‚3.4712152515235646e-05 trainer/global_step‚9149 % _timestamp‚1.7884372248373837e+09 _runtime‚9203.192452792’¾¹!Έ%È ' lr-AdamW/pg1‚3.4712152515235646e-05 ' lr-AdamW/pg2‚3.4712152515235646e-05 trainer/global_step‚9149 % _timestamp‚1.7884372248373837e+09 _runtime‚9203.192452792 _step‚749íWg2‰%- _wandbruntime‚9203  _runtime‚9203vÅ‹ú (Š%šP VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1221485834941440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71864193 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9203.193478979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360963343 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1262273996882432 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196985314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _step‚750 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197008420 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169944406 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19548163787264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1242722727701504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197008420 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1061815031996928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 trainer/global_step‚9149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1242722727701504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27120870684160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1088935959792640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27120927795712 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1221485860369408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19515107964440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19551269180928 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1242722701309952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169944958 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1061815003862528 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1241000942905880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1262270865097216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163978029 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1088935874546688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98080762 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163978035 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163978035 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1262273996882432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19551269180928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19548163787264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71863644 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19518211433364XD‹Æ%(‹%ŸP GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884372248381996e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360986455 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1088935959792640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27120927795712 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1221485860369408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19515107964440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19551269180928 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1242722701309952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169944958 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1061815003862528 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1241000942905880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1262270865097216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163978029 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1088935874546688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98080762 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163978035 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163978035 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1262273996882432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19551269180928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19548163787264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71863644 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19518211433364 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1221485834941440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71864193 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9203.193478979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360963343 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1262273996882432 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196985314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _step‚750 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197008420 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169944406 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19548163787264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1242722727701504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197008420 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1061815031996928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 trainer/global_step‚9149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1242722727701504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27120870684160 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361024855 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27123779384320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163995313 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1061927442861056 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19517216757540 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197006404 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1262404471499264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71871035 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1221615257370624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19520320226672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _runtime‚9204.103559504 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9149 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163995319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163995313 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1262407603297792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27123838579712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361001717 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1242854221789696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1241135577597296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169962050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1061927414726656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884372257477765e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361024855 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197029536 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1089051281440768 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1089051194110976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98091015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1262407603297792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361001717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71871577 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98091018 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361024855 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27123779384320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163995313 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1061927442861056 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19517216757540 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197006404 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1262404471499264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71871035 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1221615257370624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19520320226672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _runtime‚9204.103559504 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9149 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163995319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163995313 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1262407603297792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27123838579712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361001717 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1242854221789696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1241135577597296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169962050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1061927414726656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884372257477765e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361024855 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197029536 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1089051281440768 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1089051194110976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98091015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1262407603297792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361001717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71871577 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98091018 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170836281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198066272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19653793893888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1227955685002240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19623664679572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067435575203328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1227955710430208 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1268954331398656 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1249297432111104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19620561210648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1268951199613440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198043166 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164842229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚9199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1268951199613440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1249297405719552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72242290 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1249297405719552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1268954331398656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198066272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98593415 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362908507 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164842229 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170835702 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98593412 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1247576246212888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚754 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884372726716607e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198043166 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112PµN•¿%ŸP M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164842229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚9199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1268951199613440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1249297405719552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72242290 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1249297405719552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1268954331398656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198066272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98593415 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362908507 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164842229 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170835702 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98593412 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1247576246212888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 _step‚754 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884372726716607e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.cuãâWífrrent‚87343104 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198043166 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067435547068928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1249297432111104 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1094702566762496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19656899287552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27267078902272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362885395 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚579 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170836281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198066272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19653793893888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1227955685002240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19623664679572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067435575203328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1227955710430208 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1268954331398656 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1249297432111104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19620561210648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1268951199613440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198043166òb&×2ž%- _wandbruntime‚9251  _runtime‚9251‰9M)*'Ÿ%¤N PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170853851 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170853293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27269988839424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19655906207744 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1094817974906880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19659011614720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72250183 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198087388 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1249428926199296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚558 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19659011614720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19655906207744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27269929644032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19622670003748 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198087388 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362946907 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362946907 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98603668 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362923769 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1249428899807744 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067547957933056 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1228085082003456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1269084806015488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1094817887577088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362923769 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164859519 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1247707752007204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198064256 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1067547986067456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 _runtime‚9251.921273909 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302«YÅù/' %©N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164859513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98603665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1249428899807744 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067547957933056 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1228085082003456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1269084806015488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1094817887577088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362923769 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164859519 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1247707752007204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198064256 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1067547986067456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 _runtime‚9251.921273909 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170853851 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170853293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27269988839424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19655906207744 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1094817974906880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19659011614720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72250183 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198087388 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1249428926199296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚558 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19659011614720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19655906207744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27269929644032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19622670003748 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198087388 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362946907 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362946907 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98603668 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362923769 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 BfdamW/pg1‚3.252362587809876e-05 & lr-AdamW/pg2‚3.252362587809876e-05 trainer/global_step‚9249þ¼£Ë¬%Å _runtime‚9298.652105044 _step‚757 & lr-AdamW/pg1‚3.252362587809876e-05 & lr-AdamW/pg2‚3.252362587809876e-05 trainer/global_step‚9249 $ _timestamp‚1.788437320297068e+09õ2`Ðõ2­%- _wandbruntime‚9298  _runtime‚92983.°Ü(®%™P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165706435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1100469285849600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1255872136520704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1234425560491008 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1100469202700800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1275631534129664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19759424000512 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171726175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚9249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199101018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19729117925780 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19762529394176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171726725 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199124124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165706429 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72620113 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1073056118409728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27413167439872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19726014456856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364807447 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1255872136520704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27413112425472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364830559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1254151549519896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1275631534129664 _runtime‚9298.653516459 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1275634665914880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924HÛÉ$(¯%žP aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1234425560491008 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1100469202700800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1275631534129664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19759424000512 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171726175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚9249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199101018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19729117925780 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19762529394176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171726725 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199124124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165706429 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72620113 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1073056118409728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27413167439872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19726014456856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364807447 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1255872136520704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27413112425472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364830559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1254151549519896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1275631534129664 _runtime‚9298.653516459 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1275634665914880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165706435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1100469285849600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1255872136520704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106öCC2°%- _wandbruntime‚9299  _runtime‚9299wâé+*'±%¤N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1256003604217344 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165723719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72628055 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199122108 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1073168529273856 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚9249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165723719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1275765140531712 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1100584623747072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165723713 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171744373 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27416039472128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199145240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27416094473216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165723713 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19731226719088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99116318 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9299.565526972 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19761536314368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19761536314368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1275768272330240 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171743853 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1275768272330240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19764641721344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72627538 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199145240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1234554957492224 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364845821 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364868959 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1256003604217344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199122108 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364845821 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364868959 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19728123249956 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1256003630608896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440k½õ‰Æ²%©N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1254286184211312 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1234554932064256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1073168501139456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99116315 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1275768272330240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19764641721344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72627538 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak·Û)i ‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199145240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1234554957492224 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364845821 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364868959 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1256003604217344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199122108 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364845821 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364868959 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19728123249956 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1256003630608896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1256003604217344 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165723719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72628055 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199122108 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1073168529273856 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚9249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165723719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1275765140531712 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1100584623747072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165723713 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171744373 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27416039472128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199145240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27416094473216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165723713 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19731226719088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99116318 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9299.565526972 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19761536314368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19761536314368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1275768272330240 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171743853 BXå½ÌÀ%Æ & lr-AdamW/pg1‚3.145105659913143e-05 & lr-AdamW/pg2‚3.145105659913143e-05 trainer/global_step‚9299 % _timestamp‚1.7884373679214036e+09 _runtime‚9346.276486097 _step‚761ù¦æs2Á%- _wandbruntime‚9346  _runtime‚9346"fº(Â%™P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19868159500800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366752611 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1260729981723796 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884373679222324e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166570635 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1282315000431104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1240895385123840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200158870 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚89440256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚762 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1262446814538752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366729499 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200181976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61305856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1282311868645888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27559061881856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166570635 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19834571171988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚89440256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚762 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1262446814538752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366729499 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200181976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61305856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1282311868645888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27559061881856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166570635 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19834571171988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1260729981723796 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884373679222324e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166570635 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1282315000431104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1240895385123840úêt¸Ù2Ä%- _wandbruntime‚9347  _runtime‚9347ž*Ì)'Å%£N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366767873 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166587919 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99628968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99628965 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1078789044345856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1282448606846464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19867166420992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366791011 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1260861487518320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166587913 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1262578335018496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1282448606846464 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19870271827968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200203092 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1106350954507264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1241024807553024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19867166420992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19833576496164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1282445475047936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172633572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166587919 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1078789072480256 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚763ŠÐno.'Æ%¨N F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19867166420992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19833576496164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1282445475047936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172633572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166587919 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1078789072480256 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _step‚763 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366767873 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166587919 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99628968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99628965 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1078789044345856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1282448606846464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19867166420992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366791011 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1260861487518320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166587913 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1262578335018496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1282448606846464 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19870271827968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200203092 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1106350954507264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1241024807553024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1288992203162112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19973789607424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19936920949272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19970684213760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173506902 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884374150841842e+09 _runtime‚9393.439443981 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1084297176688128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73375537 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1269021518948352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167434835 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1269021545339904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100131362 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1112002291221504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1269021518948352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1112002376467456 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1084297204822528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1267302156133912 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167434835 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19973789607424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100131365 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27705114533376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201239828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19970684213760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19940024418196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368651551 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _step‚766 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167434829 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201239828 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167434829 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1288992203162112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 trainer/global_step‚9349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368674663 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201216722 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1288995334947328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1267305285030804 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73374986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73374986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1288992203162112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19973789607424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19936920949272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19970684213760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173506902 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884374150841842e+09 _runtime‚9393.439443981 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1084297176688128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73375537 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1269021518948352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167434835 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1269021545339904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100131362 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1112002291221504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1269021518948352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1112002376467456 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1084297204822528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1267302156133912 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167434835 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19973789607424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100131365 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27705114533376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201239828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19970684213760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984þž¯2Ö%- _wandbruntime‚9394  _runtime‚9394•y¼ˆ,'×%¦N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368713063 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173524563 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201260944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173524024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73382945 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1267433661928228 % _timestamp‚1.7884374159783728e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19972796527616 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1112117712273408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167452113 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1269153013036544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1289128941362688 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1084409615686656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9394.333891534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1112117624943616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368713063 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1084409587552256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 368689925 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1267436790825328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19975901934592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167452119 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 trainer/global_step‚9349 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚767 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247494657613824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1247494632185856 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201237812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73382409 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19975901934592 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1289128941362688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032ŸeÏÝ7Ø%«N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1112117624943616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368713063 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1084409587552256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 368689925 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1267436790825328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19975901934592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167452119 SFDeviceStatsMonitorˆªIú".on_train_batch_end/inactive_split_bytes.all.current‚87329792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 trainer/global_step‚9349 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚767 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247494657613824 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1247494632185856 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201237812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73382409 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19975901934592 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1289128941362688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368713063 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173524563 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201260944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173524024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73382945 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1267433661928228 % _timestamp‚1.7884374159783728e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19972796527616 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1112117712273408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167452113 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1269153013036544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1289128941362688 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1084409615686656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9394.333891534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132ÿ{°”2Ù%- _wandbruntime‚9394  _runtime‚9394™d WUÚ%Ï trainer/global_step‚9349 1loss/FBt4/assignment_loss‚0.27614022315447073 /loss/FRt1/detection_loss‚1.4128047613885955 /loss/FBt2/detection_loss‚2.2979608641730414 /loss/FRt2/detection_loss‚2.5349134020507336 /loss/FRt4/detection_loss‚2.8021144887496687 2loss/SRqqt1/assignment_loss‚1.1463997559293229 1loss/SRqqt4/detection_loss‚2.1984783069389624 /loss/FBt1/detection_loss‚0.5971659012397544 epoch‚7 % _timestamp‚1.7884374159803083e+09 .loss/FBt3/assignment_loss‚0.24200730260262 /loss/FRt2/assignment_loss‚3.260090009868145 /loss/FRt3/assignment_loss‚2.729401619538017 0loss/FRt4/assignment_loss‚2.9020937270131606 1loss/SRqqt2/assignment_loss‚1.382577805834658 1loss/SRqqt3/assignment_loss‚1.384108679862693 .loss/FBt3/detection_loss‚2.352459873471941 -loss/FBt4/detection_loss‚2.24832726418972 0loss/FRt1/assignment_loss‚1.1160436851390894 .loss/FRt3/detection_loss‚3.314322204693504 0loss/SRqqt3/detection_loss‚1.953417307585478 $loss/total_loss‚41.0897913267491 _step‚768 2loss/SRqqt4/assignment_loss‚1.3402275192982904 2loss/FBt1/assignment_loss‚0.000361469324161787 _runtime‚ 9394.33530404 1loss/FBt2/assignment_loss‚0.37893378263458366 0loss/SRqqt1/detection_loss‚0.921384950157474 1loss/SRqqt2/detection_loss‚2.2980564219110153Œ7 ØZÛ%Ô % _timestamp‚1.7884374159803083e+09 .loss/FBt3/assignment_loss‚0.24200730260262 /loss/FRt2/assignment_loss‚3.260090009868145 /loss/FRt3/assignment_loss‚2.729401619538017 0loss/FRt4/assignment_loss‚2.9020937270131606 1loss/SRqqt2/assignment_loss‚1.382577805834658 1loss/SRqqt3/assignment_loss‚1.384108679862693 .loss/FBt3/detection_loss‚2.352459873471941 -loss/FBt4/detection_loss‚2.24832726418972 0loss/FRt1/assignment_loss‚1.1160436851390894 .loss/FRt3/detection_loss‚3.314322204693504 0loss/SRqqt3/detection_loss‚1.953417307585478 $loss/total_loss‚41.0897913267491 _step‚768 2loss/SRqqt4/assignment_loss‚1.3402275192982904 2loss/FBt1/assignment_loss‚0.000361469324161787 _runtime‚ 9394.33530404 1loss/FBt2/assignment_loss‚0.37893378263458366 0loss/SRqqt1/detection_loss‚0.921384950157474 1loss/SRqqt2/detection_loss‚2.2980564219110153 trainer/global_step‚9349 1loss/FBt4/assignment_loss‚0.27614022315447073 /loss/FRt1/detection_loss‚1.4128047613885955 /loss/FBt2/detection_loss‚2.2979608641730414 /loss/FRt2/detection_loss‚2.5349134020507336 /loss/FRt4/detection_loss‚2.8021144887496687 2loss/SRqqt1/assignment_loss‚1.1463997559293229 1loss/SRqqt4/detection_loss‚2.1984783069389624 /loss/FBt1/detection_loss‚0.5971659012397544 epoch‚7€ÜÒWáÜ%‚(:’ ²ÇåÔðŽÉ¶ gpu.0.gpu‚84.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚300.154' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.03079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01,B€kÝ%‚(:à ²ÇåÔ«ë·% disk./.usageGB‚286.16883850097656 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375$ memory_percent‚98.67017269134521 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚ 3.163098e+06* disk./.usagePercent‚19.205030863449103 disk.sda.in‚0.203125" disk.nvme0n1.out‚ 69082.3046875 network.sent‚ 2.3580521e+07 disk.sda1.in‚0 disk.dm-1.out‚ 351.93359375 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2952.296875 cpu‚25.26932164681857 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 69082.3046875 disk.sda3.out‚ 351.93359375 proc.cpu.threads‚17 disk.dm-1.in‚0.203125 disk.dm-0.out‚ 69082.3046875 disk.drbd1716.in‚0 disk.rbd1.out‚4.90625 disk.dm-0.in‚ 466.90234375 disk.sda.out‚ 351.93359375* proc.memory.percent‚18.019390106201172 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.rbd1.in‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 217.87890625ø G‡’Þ%‚(:‡ ÁÇåÔ¬ö´ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚142.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.447 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X=â1jß%‚(:ß ÁÇåÔç”ß· disk.sda3.out‚ 352.54296875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.availableMB‚217.625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 466.90234375 disk.dm-0.out‚69140.82421875 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2952.296875 disk.sda2.in‚0* disk./.usagePercent‚19.206860805303855 disk.drbd2265.in‚0 disk.dm-1.out‚ 352.54296875 disk.rbd1.out‚4.90625" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.out‚0$ memory_percent‚98.67172241210938* proc.memory.percent‚18.019390106201172 network.sent‚ 2.3689606e+07 disk.drbd2266.out‚0% disk.nvme0c0n1.out‚69140.82421875 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 466.90234375 network.recv‚ 3.170063e+06% disk./.usageGB‚286.19610595703125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 352.54296875 disk.drbd2265.out‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚69140.82421875 disk.dm-1.in‚0.203125 cpu‚25.268973902705415 proc.cpu.threads‚17/LV“à%‚(:ˆ ÐÇåÔŸŽšµ gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚267.303' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4606 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å6ûªwá%‚(:ì ÐÇåÔôêã· disk.dm-1.out‚ 353.05859375 proc.cpu.threads‚17* disk./.usagePercent‚19.203932079112146 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2080.out‚0 disk.drbd2265.in‚0 cpu‚25.268626498565293 network.recv‚ 3.173542e+06( disk.nvme0c0n1.out‚69210.99560546875 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2952.296875 network.sent‚ 2.3694883e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚4.90625 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.sda.in‚0.203125$ memory_percent‚98.67169857025146 disk.rbd0.out‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚69210.99560546875 disk.sda1.in‚0 disk.sda.out‚ 353.05859375( proc.memory.availableMB‚ 217.62890625* proc.memory.percent‚18.019390106201172$ disk./.usageGB‚286.1524658203125& disk.nvme0n1.out‚69210.99560546875 disk.sda3.out‚ 353.05859375 disk.drbd1716.in‚0L?2¹2â%- _wandbruntime‚9440  _runtime‚9440?‹‘/Éã%à ' lr-AdamW/pg1‚2.9350423682968075e-05 ' lr-AdamW/pg2‚2.9350423682968075e-05 trainer/global_step‚9399 % _timestamp‚1.7884374622942045e+09 _runtime‚9440.649280607 _step‚769ÈPã Îä%È ' lr-AdamW/pg1‚2.9350423682968075e-05 ' lr-AdamW/pg2‚2.9350423682968075e-05 trainer/global_step‚9399 % _timestamp‚1.7884374622942045e+09 _runtime‚9440.649280607 _step‚769·þX£2å%- _wandbruntime‚9440  _runtime‚9440–ŸŒu!(æ%›P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 370596715 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73754268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1089917719894528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20042374195480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚770 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202274574 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174398838 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1273880588337812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1273877459440920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168299029 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1295672537678336 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 _runtime‚9440.650160929 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202274574 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1275596223357952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20076314320384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1253835110673408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27851508152320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1295672537678336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168299035 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100644012 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1117769170935296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788437462294916e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20076314320384 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168299035 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1253835085245440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202297680 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100644015 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20079419714048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73754823 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1295675669463552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1089917748028928 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1295672537678336 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 _runtime‚9440.650160929 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202274574 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1275596223357952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20076314320384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1253835110673408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27851508152320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1295672537678336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168299035 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100644012 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1117769170935296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788437462294916e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20076314320384 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168299035 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1253835085245440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202297680 YKDeviceStatsMonitor.on_train_batch_start/inactive_¿ÚÐ)›split.large_pool.allocated‚ 100644015 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20079419714048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73754823 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1295675669463552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1089917748028928 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73754268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1089917719894528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20042374195480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚770 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202274574 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174398838 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1273880588337812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1273877459440920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168299029 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106‚åtÏ2è%- _wandbruntime‚9441  _runtime‚9441õN +'é%¥N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1275727717446144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1117884518620672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174415965 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168316313 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1117884603853312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 $ _timestamp‚1.788437463188284e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168316319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1253964482246656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202318796 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20081532041216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202318796 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1274012094132336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1253964507674624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27854387862016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1295809275878912 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 370635115 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20047586457712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 370611977 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20078426634240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27854444960256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73762231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168316313 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1090030158893056 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1090030130758656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370635115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168316319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73761700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚9441.543833968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1275727717446144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174416499 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202295664 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878-Eñ•0'ê%ªN L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174415965 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168316313 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1117884603853312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 $ _timestamp‚1.788437463188284e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168316319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1253964482246656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202318796 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20081532041216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202318796 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1274012094132336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1253964507674624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27854387862016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1295809275878912 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 370635115 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20047586457712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 370611977 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20078426634240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27854444960256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73762231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168316313 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1090030158893056 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1090030130758656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370635115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168316319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73761700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 _runtime‚9441.543833968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1275727717446144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174416499 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202295664 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1275727717446144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1117884518620672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 BY`ì%Ú 1loss/SRqqt2/detection_loss‚2.2430540059621515 /loss/FBt1/detection_loss‚0.6618355377877672 /loss/FBt3/detection_loss‚2.1690907099030237 % _timestamp‚1.7884374631902475e+09 /loss/FRt1/assignment_loss‚1.575514669077737 0loss/FRt3/assignment_loss‚2.7835576666726007 0loss/FRt4/assignment_loss‚3.3457691421111426 2loss/SRqqt4/assignment_loss‚1.3536686620791443 1loss/FBt2/assignment_loss‚0.17382355937017843 0loss/FRt2/assignment_loss‚3.3318877609408633 2loss/SRqqt1/assignment_loss‚0.9663252455778839 /loss/FBt2/detection_loss‚2.8212907910346985 _runtime‚9441.545181081 1loss/SRqqt1/detection_loss‚0.8249910188566768 0loss/SRqqt4/detection_loss‚2.419742424041033 trainer/global_step‚9399 %loss/total_loss‚40.89367636515583 /loss/FBt4/detection_loss‚1.9135355283232296 /loss/FRt1/detection_loss‚1.5159318353182503 -loss/FRt2/detection_loss‚2.25340137797959 1loss/FBt4/assignment_loss‚0.12925179482108967 _step‚772 2loss/FBt1/assignment_loss‚0.014895438708076279 1loss/FBt3/assignment_loss‚0.14292832050205106 /loss/FRt3/detection_loss‚2.3602167083157433 /loss/FRt4/detection_loss‚3.1857903972268105 1loss/SRqqt3/detection_loss‚1.8036176682102913 epoch‚7 2loss/SRqqt2/assignment_loss‚1.5667388056619809 2loss/SRqqt3/assignment_loss‚1.3368172966738177-RÎÕeí%ß 1loss/SRqqt2/detection_loss‚2.2430540059621515 /loss/FBt1/detection_loss‚0.6618355377877672 /loss/FBt3/detection_loss‚2.1690907099030237 % _timestamp‚1.7884374631902475e+09 /loss/FRt1/assignment_loss‚1.575514669077737 0loss/FRt3/assignment_loss‚2.7835576666726007 0loss/FRt4/assignment_loss‚3.3457691421111426 2loss/SRqqt4/assignment_loss‚1.3536686620791443 1loss/FBt2/assignment_loss‚0.17382355937017843 0loss/FRt2/assignment_loss‚3.3318877609408633 2loss/SRqqt1/assignment_loss‚0.9663252455778839 /loss/FBt2/detection_loss‚2.8212907910346985 _runtime‚9441.545181081 1loss/SRqqt1/detection_loss‚0.8249910188566768 0loss/SRqqt4/detection_loss‚2.419742424041033 trainer/global_step‚9399 %loss/total_loss‚40.89367636515583 /loss/FBt4/detection_loss‚1.9135355283232296 /loss/FRt1/detection_loss‚1.5159318353182503 -loss/FRt2/detection_loss‚2.25340137797959 1loss/FBt4/assignment_loss‚0.12925179482108967 _step‚772 2loss/FBt1/assignment_loss‚0.014895438708076279 1loss/FBt3/assignment_loss‚0.14292832050205106 /loss/FRt3/detection_loss‚2.3602167083157433 /loss/FRt4/detection_loss‚3.1857903972268105 1loss/SRqqt3/detection_loss‚1.8036176682102913 epoch‚7 2loss/SRqqt2/assignment_loss‚1.5667388056619809 2loss/SRqqt3/assignment_loss‚1.3368172966738177„½×§I“î%‚(:ˆ ßÇåԴКµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚280.443' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.0886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04?ùrï%‚(:ç ßÇåÔÜÞ׶ disk.sda2.out‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 466.90234375 network.recv‚ 3.178276e+06& disk.nvme0n1.out‚69261.30810546875 disk.dm-1.out‚ 353.7421875% disk./.usageGB‚286.17397689819336 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚ 353.7421875 disk.sda3.in‚0.203125* proc.memory.percent‚18.019390106201172 network.sent‚ 2.3700893e+07 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.rbd1.out‚4.90625( disk.nvme0c0n1.out‚69261.30810546875 disk.sda3.out‚ 353.7421875$ memory_percent‚98.67191314697266 disk.drbd1716.out‚0 disk.drbd2266.out‚0% proc.memory.availableMB‚ 217.59375! proc.memory.rssMB‚ 2952.296875* disk./.usagePercent‚19.205375705597724 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚69261.30810546875 cpu‚25.268260047226523 proc.cpu.threads‚17 disk.sda1.out‚0(Δ“ð%‚(:ˆ îÇåÔ¹Õõ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð…M›uñ%‚(:ê îÇåÔÁž–·* proc.memory.percent‚18.019390106201172 proc.cpu.threads‚17 disk.drbd2080.in‚0& disk.nvme0n1.out‚69315.24169921875$ memory_percent‚98.67188930511475 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 network.sent‚ 2.380963e+07 disk.dm-0.in‚ 466.90234375 disk.sda3.out‚ 354.11328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚4.90625 cpu‚25.26791236803164 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 354.11328125 disk.dm-1.out‚ 354.11328125( proc.memory.availableMB‚ 217.59765625* disk./.usagePercent‚19.207207439505215$ disk./.usageGB‚286.2012710571289# disk.dm-0.out‚69315.24169921875 disk.sda1.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2952.296875 network.recv‚ 3.184832e+06" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚69315.24169921875 disk.sda3.in‚0.203125 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125Hò$žò%‚(:“ ýÇåÔ醵 gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.217600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í4qó%‚(:æ ýÇåÔ۔ⷠdisk.drbd1716.out‚0 disk.drbd2080.out‚0# memory_percent‚98.6734390258789 cpu‚25.26756528433156 proc.cpu.threads‚17) disk./.usagePercent‚19.20858322399422 disk.sda3.out‚ 354.72265625 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.out‚ 354.72265625 disk.rbd1.in‚0 disk.rbd1.out‚4.90625! proc.memory.rssMB‚ 2952.296875 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0 disk.sda.out‚ 354.72265625 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 network.recv‚ 3.189856e+06% proc.memory.availableMB‚ 217.34375 disk.drbd2080.in‚0 disk.drbd2265.out‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚69361.74560546875 network.sent‚ 2.3816026e+07$ disk./.usageGB‚286.2217712402344# disk.dm-0.out‚69361.74560546875 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚69361.74560546875 disk.sda2.out‚0* proc.memory.percent‚18.019390106201172(|!P2ô%- _wandbruntime‚9487  _runtime‚9487‰.ù´Çõ%Á & lr-AdamW/pg1‚2.832279781452323e-05 & lr-AdamW/pg2‚2.832279781452323e-05 trainer/global_step‚9449 % _timestamp‚1.7884375094930575e+09 _runtime‚9487.848126164 _step‚773ÊÈp\ö%Æ _runtime‚9487.848126164 _step‚773 & lr-AdamW/pg1‚2.832279781452323e-05 &GL1“p lr-AdamW/pg2‚2.832279781452323e-05 trainer/global_step‚9449 % _timestamp‚1.7884375094930575e+09…\ ¸2÷%- _wandbruntime‚9487  _runtime‚9487,Kz"(ø%œP TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169163229 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚9449 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27997506477568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1260304935306240 _step‚774 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1123535769578496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20185049820672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20181944427008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175287323 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203355532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20181944427008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74130111 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1302352872194560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203332426 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20147827441688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169163235 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1123535854824448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚9487.849127554 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1302356003979776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1302356003979776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101156665 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1260304960734208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1280455891644820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169163235 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884375094938424e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203332426 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175286773 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1282170927767552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101156662 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1280452762747928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20150930910612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169163229 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1095538263100928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27997563589120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1282170954159104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 372518767 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302£¡U'(ù%¡P [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1123535769578496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20185049820672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20181944427008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175287323 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203355532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20181944427008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74130111 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1302352872194560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203332426 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20147827441688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169163235 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1123535854824448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚9487.849127554 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1302356003979776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1302356003979776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101156665 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1260304960734208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1280455891644820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169163235 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884375094938424e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203332426 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175286773 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1282170927767552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101156662 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1280452762747928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20150930910612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169163229 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1095538263100928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27997563589120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1282170954159104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 372518767 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169163229 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚9449 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27997506477568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1260304935306240 _step‚774†n’Ôˆ2ú%- _wandbruntime‚9488  _runtime‚9488–âØÕ,'û%¦N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1302489610395136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203376648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203353516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1095650673965056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169180513 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74138237 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74137714 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372557167 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175304629 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169180519 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1302489610395136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1282302448247296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372557167 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20149936234788 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864.A¤œ†ü%«N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20184056740864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1123651114474496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28000440509440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101166918 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372534029 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FxµÁ«.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203353516 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20187162147840 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1280587397439344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372557167 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20149936234788 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1302489610395136 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203376648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203353516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1095650673965056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169180513 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74138237 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74137714 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372557167 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175304629 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169180519 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1302489610395136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1282302448247296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1309033206710784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1309036338496000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204413384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1309036338496000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20253280687896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170027429 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204390278 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1266774785367040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204390278 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1129302528628736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _step‚778 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20287574533632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 trainer/global_step‚9499 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1288745658568704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1288745632177152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1266774810795008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374440819 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170027435 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28143722321408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101669315 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204413384 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28143781530112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20256384156820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374417707 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74507335 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170027435 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101669312 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1129302615971840 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1101158834441728 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1101158806307328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104¯µ0׋& P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _step‚778 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20287574533632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 trainer/global_step‚9499 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1288745658568704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1288745632177152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1266774810795008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374440819 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170027435 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28143722321408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101669315 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204413384 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28143781530112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20256384156820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374417707 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74507335 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170027435 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101669312 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1129302615971840 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1101158834441728 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1101158806307328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1309033206710784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1309036338496000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204413384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1309036338496000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20253280687896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170027429 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204390278 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1266774785367040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204390278 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1129302528628736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ŠÚj[s2Œ&- _wandbruntime‚9536  _runtime‚9536[¬•æ,'&¦N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374456081 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204434500 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1288877152656896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204411368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884375577168677e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1288877152656896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176194402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚9536.072459655 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374479219 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1309169944911360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20292792254464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20258492950128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1309169944911360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1288877126265344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9499 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74515364 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74514837 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20255389480996 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170044719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1101271245305856 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170044719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1266904182368256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20292792254464 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1266904207796224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20289686847488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1309166813112832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101679568 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1129417870313472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1287159571849252 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374479219 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28146653142016 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170044713 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204434500 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1129417957643264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28146712337408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170044719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1101271245305856 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170044719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1266904182368256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20292792254464 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1266904207796224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20289686847488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1309166813112832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101679568 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1129417870313472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1287159571849252 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374479219 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28146653142016 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170044713 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204434500 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1129417957643264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28146712337408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884375577168677e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1288877152656896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176194402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚9536.072459655 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374479219 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1309169944911360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20292792254464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20258492950128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1309169944911360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1288877126265344‹®Â92&- _wandbruntime‚9536  _runtime‚9536Y· Q^&Ø 2loss/FBt1/assignment_loss‚0.007071873126710196 1loss/SRqqt2/detection_loss‚2.2937039028514516 _runtime‚9536.073814653 0loss/FRt1/assignment_loss‚1.3764857823650043 /loss/FRt1/detection_loss‚1.3370063889564738 1loss/SRqqt3/detection_loss‚1.6721162938379817 /loss/FBt4/detection_loss‚1.8542276918888092 /loss/FRt4/assignment_loss‚3.898061167809271 2loss/SRqqt1/assignment_loss‚0.9875041006175294 1loss/FBt3/assignment_loss‚0.20324045621268283 /loss/FRt3/detection_loss‚2.6008166074752808 1loss/SRqqt1/detection_loss‚0.9257759184745903 /loss/FBt1/detection_loss‚0.8401621386253585 % _timestamp‚1.7884375577187953e+09 0loss/SRqqt3/assignment_loss‚1.37727343264435 _step‚780 .loss/FRt2/detection_loss‚2.388614898218828 /loss/FBt2/detection_loss‚2.3382247537374496 %loss/total_loss‚42.48253072810015 trainer/global_step‚9499 0loss/FRt2/assignment_loss‚3.6747728726443123 0loss/FRt3/assignment_loss‚3.1501390586296716 1loss/FBt2/assignment_loss‚0.18242135541368043 1loss/SRqqt4/detection_loss‚2.4137313750482376 1loss/FBt4/assignment_loss‚0.41370804637780423 -loss/FRt4/detection_loss‚2.75384747405206 2loss/SRqqt2/assignment_loss‚1.9362336519089611 2loss/SRqqt4/assignment_loss‚1.4095478908444243 /loss/FBt3/detection_loss‚2.4478435963392258 epoch‚7„2c‘&Ý 2loss/SRqqt2/assignment_loss‚1.9362336519089611 2loss/SRqqt4/assignment_loss‚1.4095478908444243 /loss/FBt3/detection_loss‚2.4478435963392258 epoch‚7 2loss/FBt1/assignment_loss‚0.007071873126710196 1loss/SRqqt2/detection_loss‚2.2937039028514516 _runtime‚9536.073814653 0loss/FRt1/assignment_loss‚1.3764857823650043 /loss/FRt1/detection_loss‚1.3370063889564738 1loss/SRqqt3/detection_loss‚1.6721162938379817 /loss/FBt4/detection_loss‚1.8542276918888092 /loss/FRt4/assignment_loss‚3.898061167809271 2loss/SRqqt1/assignment_loss‚0.9875041006175294 1loss/FBt3/assignment_loss‚0.20324045621268283 /loss/FRt3/detection_loss‚2.6008166074752808 1loss/SRqqt1/detection_loss‚0.9257759184745903 /loss/FBt1/detection_loss‚0.8401621386253585 % _timestamp‚1.7884375577187953e+09 0loss/SRqqt3/assignment_loss‚1.37727343264435 _step‚780 .loss/FRt2/detection_loss‚2.388614898218828 /loss/FBt2/detection_loss‚2.3382247537374496 %loss/total_loss‚42.48253072810015 trainer/global_step‚9499 0loss/FRt2/assignment_loss‚3.6747728726443123 0loss/FRt3/assignment_loss‚3.1501390586296716 1loss/FBt2/assignment_loss‚0.18242135541368043 1loss/SRqqt4/detection_loss‚2.4137313750482376 1loss/FBt4/assignment_loss‚0.41370804637780423 -loss/FRt4/detection_loss‚2.75384747405206ŒüÔm’&‚(:’ ¹ÈåÔ«âµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚285.334' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.06679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üp"j“&‚(:ß ¹ÈåÔΖ‰· disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚69597.56103515625 disk.sda1.out‚0 disk.drbd2080.in‚0 network.recv‚ 3.20878e+06* disk./.usagePercent‚19.215152377322827 disk.drbd2266.out‚0# disk.dm-0.out‚69597.56103515625 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375 network.sent‚ 2.3942212e+07 disk.rbd1.in‚0 disk.rbd1.out‚4.90625 disk.sda3.in‚0.203125& disk.nvme0n1.out‚69597.56103515625$ memory_percent‚98.67401123046875" proc.memory.availableMB‚217.25$ disk./.usageGB‚286.3196563720703 disk.dm-1.out‚ 356.703125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚18.019390106201172 cpu‚25.266204416019164 proc.cpu.threads‚17 disk.sda3.out‚ 356.703125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 356.703125 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 466.90234375! proc.memory.rssMB‚ 2952.296875Qž”&‚(:“ ÈÈåÔŸ ‰µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚287.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.560199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñçq•&‚(:æ ÈÈåÔ•¨Ä· disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 cpu‚25.26583735057944 disk.sda1.in‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.dm-1.out‚ 357.09765625 disk.sda2.out‚0 disk.dm-0.in‚ 466.90234375 disk.drbd2080.in‚0 disk.sda2.in‚0) disk./.usagePercent‚19.21588660693103 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0$ memory_percent‚98.67262840270996' proc.memory.availableMB‚ 217.4765625 disk.drbd2266.in‚0 network.recv‚ 3.213236e+06 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 357.09765625 proc.cpu.threads‚17$ disk./.usageGB‚286.3305969238281 disk.sda.out‚ 357.09765625 disk.nvme0n1.in‚ 466.90234375 disk.rbd1.out‚4.90625" disk.nvme0c0n1.in‚ 466.90234375" disk.dm-0.out‚69657.2939453125% disk.nvme0n1.out‚69657.2939453125 network.sent‚ 2.4049173e+07' disk.nvme0c0n1.out‚69657.2939453125 disk.sda3.in‚0.203125 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2952.296875* proc.memory.percent‚18.019390106201172¤üœ–&‚(:’ ×ÈåÔ΀úµ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.857' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.37140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hªÊu—&‚(:ê ×ÈåÔþ•á· disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 network.sent‚ 2.4055171e+07 network.recv‚ 3.217836e+06% disk./.usageGB‚286.36050033569336 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 357.90234375 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0' disk.nvme0c0n1.out‚69705.4580078125" disk.dm-0.out‚69705.4580078125 disk.sda3.in‚0.203125* proc.memory.percent‚18.019390106201172 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 357.90234375$ memory_percent‚98.67255687713623! proc.memory.rssMB‚ 2952.296875 cpu‚25.265524751004385* disk./.usagePercent‚19.217893449991937% disk.nvme0n1.out‚69705.4580078125 disk.dm-1.out‚ 357.90234375 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125( proc.memory.availableMB‚ 217.48828125 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚4.90625 disk.drbd2080.out‚0ævï]2˜&- _wandbruntime‚9583  _runtime‚9583ÜÒw«É™&à trainer/global_step‚9549 % _timestamp‚1.7884376047137096e+09 _runtime‚9583.068744456 _step‚781 ' lr-AdamW/pg1‚2.6313997990731275e-05 ' lr-AdamW/pg2‚2.6313997990731275e-05j>š™Îš&È ' lr-AdamW/pg1‚2.6313997990731275e-05 ' lr-AdamW/pg2‚2.6313997990731275e-05 trainer/global_step‚9549 % _timestamp‚1.7884376047137096e+09 _runtime‚9583.068744456 _step‚781’v2›&- _wandbruntime‚9583  _runtime‚9583]²èiŽœ&œP TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177065567 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1135068849788416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚9549 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1295320362978304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 1708916 ßó”%29 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1106779377648128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚782 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1273244660855808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1295320362978304 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102181965 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74883602 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28289559483392 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376339759 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205471236 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315713541227008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1293606498258836 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170891629 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20393204640256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205448130 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1273244635427840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚574 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20393204640256 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚577 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205471236 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884376047144341e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1315716673012224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315713541227008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74883028 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _runtime‚9583.069706322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1295320336586752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28289500274688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1293603369361944 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315716673012224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1135068937131520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205448130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20396310033920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20361837403028 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20358733934104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878.—R*'(&¡P A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1273244635427840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚574 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20393204640256 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚577 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205471236 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884376047144341e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1315716673012224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315713541227008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74883028 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _runtime‚9583.069706322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1295320336586752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28289500274688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1293603369361944 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315716673012224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1135068937131520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205448130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20396310033920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20361837403028 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20358733934104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177065567 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1135068849788416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚9549 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1295320362978304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170891629 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1106779377648128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚782 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1273244660855808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1295320362978304 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102181965 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74883602 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28289559483392 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376339759 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205471236 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315713541227008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1293606498258836 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170891629 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20393204640256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205448130Ž˜ÝÈ2ž&- _wandbruntime‚9583  _runtime‚9583ÂÊï©,'Ÿ&¦N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20395316954112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74890784 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1293734875156260 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1315847147629056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1295451857066496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1106891760377856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61292544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20398422361088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1135184250531328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚9549 % _timestamp‚1.7884376056285207e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1293738004053360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376378133 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170908919 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102192218 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1295451830674944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376378133 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170908913 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102192215 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20398422361088 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177082460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1315850279427584 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20363946196336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205469220 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170908913 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1295451857066496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20360842727204 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1295451830674944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170908919 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20395316954112 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376401271 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376401271 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0}b½FÄ  &«N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚89426944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1135184161104384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20395316954112 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376401271 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376401271 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20395316954112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74890784 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1293734875156260 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1315847147629056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1295451857066496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1106891760377856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 JDDeviceStatsè2²ŠmMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61292544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20398422361088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1135184250531328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚9549 % _timestamp‚1.7884376056285207e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1293738004053360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376378133 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170908919 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102192218 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1295451830674944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376378133 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170908913 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102192215 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20398422361088 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177082460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1315850279427584 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20363946196336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205469220 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170908913 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1295451857066496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20360842727204 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1295451830674944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170908919 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20467290649236 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206505982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206529088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1322397007528448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177952962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177953521 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20464187180312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1301895040996352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102694615 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171755829 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102694612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206505982 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28435206176256 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1300178672668952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1322393875743232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1301895067387904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456l2+±& P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102694615 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171755829 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102694612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206505982 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28435206176256 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1300178672668952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1322393875743232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1301895067387904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1301895040996352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1140835127030784 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1140835039687680 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1279714510916608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20467290649236 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206505982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206529088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1322397007528448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177952962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177953521 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20464187180312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1301895040996352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738’ë!q2²&- _wandbruntime‚9631  _runtime‚9631êð+'³&¥N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206527072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20500947060736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378323323 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171773113 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171773113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206527072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1279843882489856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1112512331718656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1112512303584256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20466295973412 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378323323 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1302026561476096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102704868 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1300313307360368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171773119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1302026535084544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20504052467712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 _step‚787 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20500947060736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 $ _timestamp‚1.788437653355174e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1302026535084544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177971225 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177970679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206550204 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20504052467712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚9631.711157345 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1322530613943808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1322527482145280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102704865 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1300310178463268 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378300185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20466295973412 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378323323 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1302026561476096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102704868 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1300313307360368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171773119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1302026535084544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20504052467712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 _step‚787 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20500947060736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 $ _timestamp‚1.788437653355174e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1302026535084544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177971225 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177970679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206550204 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20504052467712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚9631.711157345 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1322530613943808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1322527482145280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102704865 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1300310178463268 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378300185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378323323 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171773113 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171773113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2“8jKk2µ&- _wandbruntime‚9631  _runtime‚9631dRh¹]¶&× _runtime‚ 9631.71231523 _step‚788 /loss/FRt2/assignment_loss‚2.595836999018987 1loss/SRqqt4/assignment_loss‚1.399277135344415 3loss/FBt1/assignment_loss‚0.0009315940763211393 epoch‚7 trainer/global_step‚9599 % _timestamp‚1.7884376533573194e+09 /loss/FRt1/detection_loss‚1.3268090561032295 .loss/FBt3/detection_loss‚2.379444713393847 0loss/FRt4/assignment_loss‚4.2439876264995995 2loss/SRqqt1/assignment_loss‚0.7523617734732007 1loss/FBt2/assignment_loss‚0.18781559197304842 1loss/FBt4/assignment_loss‚0.12122575546142317 1loss/SRqqt2/detection_loss‚2.3316916900296367 1loss/SRqqt2/assignment_loss‚1.535302565342957 /loss/FRt3/detection_loss‚2.3231069026766598 1loss/SRqqt1/detection_loss‚0.8375136735298301 .loss/FBt2/detection_loss‚2.187288023531437 %loss/total_loss‚41.00246234148123 2loss/SRqqt3/assignment_loss‚1.3784693277899236 .loss/FRt2/detection_loss‚2.897080608208974 .loss/FRt4/detection_loss‚3.024134691114779 /loss/FBt4/detection_loss‚2.1047466221977684 0loss/FRt1/assignment_loss‚1.3552951284817287 0loss/FRt3/assignment_loss‚3.2735289335250854 1loss/FBt3/assignment_loss‚0.14503990307881529 /loss/FBt1/detection_loss‚0.6022679811447031 1loss/SRqqt3/detection_loss‚1.5386683671247392 1loss/SRqqt4/detection_loss‚2.4606376783601167â7†b·&Ü /loss/FBt4/detection_loss‚2.1047466221977684 0loss/FRt1/assignment_loss‚1.3552951284817287 0loss/FRt3/assignment_loss‚3.2735289335250854 1loss/FBt3/assignment_loss‚0.14503990307881529 /loss/FBt1/detection_loss‚0.6022679811447031 1loss/SRqqt3/detection_loss‚1.5386683671247392 1loss/SRqqt4/detection_loss‚2.4606376783601167 _runtime‚ 9631.71231523 _step‚788 /loss/FRt2/assignment_loss‚2.595836999018987 1loss/SRqqt4/assignment_loss‚1.399277135344415 3loss/FBt1/assignment_loss‚0.0009315940763211393 epoch‚7 trainer/global_step‚9599 % _timestamp‚1.7884376533573194e+09 /loss/FRt1/detection_loss‚1.3268090561032295 .loss/FBt3/detection_loss‚2.379444713393847 0loss/FRt4/assignment_loss‚4.2439876264995995 2loss/SRqqt1/assignment_loss‚0.7523617734732007 1loss/FBt2/assignment_loss‚0.18781559197304842 1loss/FBt4/assignment_loss‚0.12122575546142317 1loss/SRqqt2/detection_loss‚2.3316916900296367 1loss/SRqqt2/assignment_loss‚1.535302565342957 /loss/FRt3/detection_loss‚2.3231069026766598 1loss/SRqqt1/detection_loss‚0.8375136735298301 .loss/FBt2/detection_loss‚2.187288023531437 %loss/total_loss‚41.00246234148123 2loss/SRqqt3/assignment_loss‚1.3784693277899236 .loss/FRt2/detection_loss‚2.897080608208974 .loss/FRt4/detection_loss‚3.024134691114779”›\Cʸ&‚(:’ ¢ÉåÔå佶 gpu.0.gpu‚48.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚138.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.656000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z½KÂp¹&‚(:å ¢ÉåÔÕÕõ¶ disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2952.296875 cpu‚25.26383196443194' disk.nvme0c0n1.out‚69993.5498046875 disk.drbd2265.in‚0 disk.dm-1.out‚ 360.32421875 disk.drbd2266.out‚0 disk.rbd1.in‚0$ memory_percent‚98.67541790008545( proc.memory.availableMB‚ 217.01953125 proc.cpu.threads‚17 network.recv‚ 3.240868e+06 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0* disk./.usagePercent‚19.224470027539038 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 360.32421875 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda3.out‚ 360.32421875 network.sent‚ 2.418757e+07 disk.drbd2265.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚69993.5498046875 disk.rbd1.out‚4.90625" disk.dm-0.out‚69993.5498046875 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375 disk.dm-1.in‚0.203125* proc.memory.percent‚18.019390106201172" disk./.usageGB‚286.45849609375 disk.drbd2080.in‚0 disk.sda3.in‚0.203125Tm“º&‚(:ˆ ±ÉåÔÓ†¶µ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚170.643' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^³žMs»&‚(:è ±ÉåÔТ·* disk./.usagePercent‚19.225969463667084$ memory_percent‚97.95827865600586" disk.dm-0.out‚70066.2060546875 disk.rbd1.out‚4.90625 disk.dm-0.in‚ 466.90234375 cpu‚25.263498902197096 network.recv‚ 3.247657e+06 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 360.89453125 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda.out‚ 360.89453125 disk.rbd0.out‚0 disk.sda3.in‚0.203125 disk.drbd2265.out‚0* proc.memory.percent‚18.019390106201172 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0 network.sent‚ 2.4296524e+07% disk./.usageGB‚286.48083877563477 disk.rbd1.in‚0 disk.sda3.out‚ 360.89453125 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd2080.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 334.515625! proc.memory.rssMB‚ 2952.296875 proc.cpu.threads‚17 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚70066.2060546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 466.90234375% disk.nvme0n1.out‚70066.2060546875þI|5“¼&‚(:ˆ ÀÉåÔã‘¡µ gpu.0.gpu‚60.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.321' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.2642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Óq½&‚(:æ ÀÉåÔΥ㶠disk.sda.in‚0.203125 disk.sda.out‚ 361.3515625 disk.sda3.out‚ 361.3515625 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 361.3515625 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚18.019390106201172 network.sent‚ 2.4301948e+07 disk.sda3.in‚0.203125 disk.sda2.in‚0( disk.nvme0c0n1.out‚70107.02587890625 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚70107.02587890625! proc.memory.rssMB‚ 2952.296875 proc.cpu.threads‚17% disk./.usageGB‚286.52214431762695 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.90234375$ memory_percent‚97.95675277709961& proc.memory.availableMB‚ 334.765625* disk./.usagePercent‚19.228741513247837 disk.drbd2080.out‚0 disk.sda2.out‚0 network.recv‚ 3.25138e+06 disk.drbd1716.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 466.90234375 cpu‚25.26317150120687& disk.nvme0n1.out‚70107.02587890625 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.out‚4.90625 disk.drbd1716.out‚0½äÀw2¾&- _wandbruntime‚9678  _runtime‚9678QT¨iÉ¿&à _runtime‚9678.777956756 _step‚789 ' lr-AdamW/pg1‚2.4368524017123833e-05 ' lr-AdamW/pg2‚2.4368524017123833e-05 trainer/global_step‚9649 % _timestamp‚1.7884377004228814e+09¦U{WÎÀ&È trainer/global_step‚9649 % _timestamp‚1.7884377004228814e+09 _runtime‚9678.777956756 _step‚789 ' lr-AdamW/pg1‚2.4368524017123833e-05 ' lr-AdamW/pg2‚2.4368524017123833e-05•/Û„Á&- _wandbruntime¼fÁ‚9678  _runtime‚9678BõW"(Â&œP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1308469771797504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20607570247168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172620035 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178841889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1329074210259456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1118020464060928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20569640426520 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20572743895444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _runtime‚9678.778939792 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1146601010562048 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1286184335549440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1308469745405952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380206975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚594 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103207262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61305856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28580485195264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20604464853504 trainer/global_step‚9649 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178842483 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207563834 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1329077342044672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28580546501120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 % _timestamp‚1.7884377004236367e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1308469745405952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1308469771797504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1118020435926528 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1146600921121792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103207265 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1329074210259456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75634627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172620035 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20607570247168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456'X )'(Ã&¡P _step‚790 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3294746624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1306753975975960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1286184360977408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207586940 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1329074210259456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75634627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172620035 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20607570247168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1308469771797504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20607570247168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172620035 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178841889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1329074210259456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1118020464060928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20569640426520 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20572743895444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _runtime‚9678.778939792 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1146601010562048 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1286184335549440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1308469745405952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380206975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚594 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103207262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61305856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28580485195264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20604464853504 trainer/global_step‚9649 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178842483 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207563834 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1329077342044672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28580546501120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 % _timestamp‚1.7884377004236367e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1308469745405952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1308469771797504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1118020435926528 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1146600921121792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103207265 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20571749219620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1329210948460032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380222237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75642328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20609682574336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20606577167360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178859843 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1118132874925056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61292544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚570 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380245375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380245375 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207608056 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178860413 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1329210948460032 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1118132846790656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1308601265885696 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103217515 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1286313757978624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172637313 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28583400806400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 5®>Æ&«N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1329207816661504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20574852688752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207584924 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380245375 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207608056 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178860413 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1329210948460032 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1118132846790656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_s¨‰ætreams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1308601265885696 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103217515 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1286313757978624 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172637313 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28583400806400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3294746624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20606577167360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1308601239494144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1306885481770276 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚89426944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380222237 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207608056 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1306888610667376 % _timestamp‚1.7884377013288643e+09 _runtime‚9679.684546017 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1146716337024000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207584924 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1146716247597056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20571749219620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1329210948460032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380222237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75642328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20609682574336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20606577167360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178859843 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1118132874925056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61292544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚570 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380245375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 E!užÊ&‚(:“ ÏÉåÔ°‡Éµ gpu.0.gpu‚71.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚293.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.684999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÂ5~wË&‚(:ì ÏÉåÔÕ¹›·$ memory_percent‚97.95448780059814( proc.memory.availableMB‚ 335.13671875! proc.memory.rssMB‚ 2952.296875 disk.sda.in‚0.203125( disk.nvme0c0n1.out‚70153.82666015625 disk.dm-0.in‚ 466.90234375 disk.sda1.out‚0 network.recv‚ 3.255767e+06 disk.sda.out‚ 362.40234375 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.sent‚ 2.430801e+07 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.out‚0) disk./.usagePercent‚19.23073401988683 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 cpu‚25.262839939915505$ disk./.usageGB‚286.5518341064453& disk.nvme0n1.out‚70153.82666015625 disk.rbd1.out‚ 5.0546875 disk.drbd2080.out‚0# disk.dm-0.out‚70153.82666015625 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 362.40234375* proc.memory.percent‚18.019390106201172 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 362.40234375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375ì¸ü·œÌ&‚(:‘ ÞÉåÔ‹É¥µ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.335' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.866999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çûôvvÍ&‚(:ë ÞÉåÔâæᶠdisk.drbd2266.out‚0* disk./.usagePercent‚19.232255728670353 disk.dm-1.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2080.out‚0& proc.memory.availableMB‚ 334.890625& disk.nvme0n1.out‚70206.09619140625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚70206.09619140625$ memory_percent‚97.95598983764648 proc.cpu.threads‚17 network.sent‚ 2.441642e+07 network.recv‚ 3.261424e+06 disk.sda2.out‚0# disk.dm-0.out‚70206.09619140625 disk.sda1.in‚0 disk.sda.out‚ 362.75390625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.sda3.out‚ 362.75390625 disk.drbd1716.in‚0 disk.rbd1.out‚ 5.0546875 disk.dm-1.out‚ 362.75390625 disk.sda.in‚0.203125 cpu‚25.264415533405746 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.rbd0.in‚0* proc.memory.percent‚18.019580841064453$ disk./.usageGB‚286.5745086669922 disk.nvme0n1.in‚ 466.90234375 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2952.328125 disk.drbd2266.in‚0 disk.rbd0.out‚0¨œ‘«Î&‚(:… íÉåÔïÒµ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚243.815' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.763 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òæ5‘|Ï&‚(:ñ íÉåÔ€¼å¶ network.sent‚ 2.4422771e+07% disk./.usageGB‚286.56884002685547# disk.dm-0.out‚70257.66650390625 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.sda.out‚ 363.37890625 disk.sda2.out‚0 disk.rbd1.out‚ 5.0869140625" disk.nvme0c0n1.in‚ 466.90234375( proc.memory.availableMB‚ 334.79296875* proc.memory.percent‚18.019580841064453 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚70257.66650390625( disk.nvme0c0n1.out‚70257.66650390625 disk.sda3.out‚ 363.37890625 disk.rbd0.out‚0$ memory_percent‚97.95658588409424 proc.cpu.threads‚17 disk.dm-1.out‚ 363.37890625) disk./.usagePercent‚19.23187530147447 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 network.recv‚ 3.266335e+06 disk.drbd1716.in‚0 disk.drbd2080.in‚0 cpu‚25.270626639603975 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0 disk.sda.in‚0.203125! proc.memory.rssMB‚ 2952.328125¤Õíò’Ð&‚(:‡ üÉåÔ·º³µ gpu.0.gpu‚90.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚308.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00#RyÑ&‚(:î üÉåÔ¡áܶ$ disk./.usageGB‚286.6105041503906 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 363.95703125' disk.nvme0c0n1.out‚70316.7041015625 disk.drbd1716.out‚0* proc.memory.percent‚18.019580841064453* disk./.usagePercent‚19.234671415763444 disk.nvme0n1.in‚ 466.90234375" disk.nvme0c0n1.in‚ 466.90234375 disk.dm-0.in‚ 466.90234375% disk.nvme0n1.out‚70316.7041015625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.0869140625$ memory_percent‚97.95467853546143 disk.sda.in‚0.203125 disk.drbd2080.in‚0( proc.memory.availableMB‚ 335.10546875! proc.memory.rssMB‚ 2952.328125 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 3.269744e+06 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 363.95703125 network.sent‚ 2.4428104e+07 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 cpu‚25.277357080481142 disk.sda.out‚ 363.95703125" disk.dm-0.out‚70316.7041015625 disk.sda1.in‚0 disk.drbd1716.in‚0JBã%Ò&‚(:„ ‹ÊåÔÙÈŽµ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚39.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O8Œ‚wÓ&‚(:ì ‹ÊåÔ¯¨õ¶ disk.drbd1716.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2952.328125 proc.cpu.threads‚17 disk.dm-1.out‚ 364.53515625* proc.memory.percent‚18.019580841064453 cpu‚25.28368188537651 network.recv‚ 3.274073e+06$ disk./.usageGB‚286.6256561279297 disk.sda.out‚ 364.53515625 disk.drbd2265.in‚0% disk.nvme0n1.out‚70368.7509765625 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚70368.7509765625 disk.drbd2265.out‚0* disk./.usagePercent‚19.235688277689448 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0& proc.memory.availableMB‚ 335.109375 network.sent‚ 2.4434234e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 5.14990234375 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 466.90234375$ memory_percent‚97.95465469360352 disk.drbd2266.in‚0 disk.dm-0.in‚ 466.90234375 disk.drbd2080.out‚0 disk.dm-1.in‚0.203125" disk.dm-0.out‚70368.7509765625 disk.sda3.out‚ 364.53515625P|þ2Ô&- _wandbruntime‚9755  _runtime‚97559í­‹*Õ&…U _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 208430876 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353901056 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚20645591811572 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚20680574667776 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 173664269 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 173664264 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18278 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚20678245061120 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1152966673824768 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 382113423 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚28672335770112 _JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚1313968614229492 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚20643264120732 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1336564748373504 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1315884173705728 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 382095140 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 179962728 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 382095140 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚76045968 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚140 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1124294473473024 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 trainer/global_step‚9671 _runtime‚9755.623116281 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1700 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚76046108 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1124294455824384 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1293322999087104 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1336562394472448 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 179962586 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1124294455824384 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1293322999087104 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1336562394472448 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 179962586 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 382095140 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚76045968 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚140 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1124294473473024 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 trainer/global_step‚9671 _runtime‚9755.623116281 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1700 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚76046108 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _step‚794 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 173677467 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2353901056 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1314054060281396 dODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚1293407972018176 % _timestamp‚1.7884377783146982e+09 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1124369750198272 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _runtime‚9756.670088405 ]HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚1336654839000576 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 382116032 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 173677467 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚20678743078400 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚20681072685056 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚20678743078400 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1153042448567296 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117769728 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚20681072685056 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 173677462 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 382134315 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1336652485099520 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18283 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚142 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚20643760572380 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚20646088263220 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚76050916 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1153042583985664 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚28672833787392 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 208456848 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1315973766315520 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 382116032 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1336654839000576 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 208456848 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚28672716017664 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1315973742021120 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚76050776 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚140 `KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚1293407948687360 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 208438570 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1124369732549632 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984ÖÝ7š)Ù&”S TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 208438570 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1124369732549632 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1315973742021120 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3296843776 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _step‚794 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 173677467 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2353901056 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1314054060281396 dODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚1293407972018176 % _timestamp‚1.7884377783146982e+09 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1124369750198272 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 _runtime‚9756.670088405 ]HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚1336654839000576 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 382116032 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 173677467 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚20678743078400 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚20681072685056 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚20678743078400 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1153042448567296 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117769728 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚20681072685056 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 173677462 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 382134315 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1336652485099520 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18283 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚142 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚20643760572380 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚20646088263220 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚76050916 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1153042583985664 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚28672833787392 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 208456848 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1315973766315520 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 382116032 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1336654839000576 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 208456848 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚28672716017664 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1315973742021120 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚76050776 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚140 `KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚1293407948687360šyq`žžÚ&‚(:“ šÊåԬݨ¶ gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚298.451' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.690200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t…ÏÏzÛ&‚(:ï šÊåÔÞ•“· disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚70421.53662109375 disk.dm-1.out‚ 364.88671875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.sda3.out‚ 364.88671875 disk.sda1.in‚0( proc.memory.availableMB‚ 335.07421875) disk./.usagePercent‚19.23676862948394 disk.sda.out‚ 364.88671875! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453 cpu‚25.28970616895665 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.drbd1716.in‚0 disk.drbd2080.out‚0# memory_percent‚97.9548692703247 network.sent‚ 2.4440103e+07 disk.rbd0.out‚0$ disk./.usageGB‚286.6417541503906 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚70421.53662109375 disk.sda.in‚0.203125 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 3.278079e+06& disk.nvme0n1.out‚70421.53662109375 disk.sda2.in‚0 disk.rbd1.out‚ 5.14990234375ñ"éÀ2Ü&- _wandbruntime‚9771  _runtime‚9771Ý"hq Ý&Ÿ¦ =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38623514771461487 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27621692419052124 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 jet/accuracy_1_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.36911237239837646 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3739675283432007 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23533636331558228 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 _runtime‚9771.181930567 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2910611033439636 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.265398770570755 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 jet/accuracy_5_of_7‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5644841194152832 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019098583608865738 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24372969567775726 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 >&Purity/*FRt3SR ¶Xùqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 +jet/accuracy_2_of_2‚0.09975467622280121 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920675754547119 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5560247302055359 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_6‚0.1538461595773697 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.535765528678894 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18278688192367554 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2689165771007538 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03784959018230438 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14166148006916046 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36782121658325195 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8173867464065552 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_3‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.026757873594760895 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 6 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.344696968793869 particle/accuracy_4_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993274211883545 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.25613197684288025 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.37217116355895996 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.225862056016922 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_4‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0494663380086422 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.897315502166748 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 particle/accuracy_7_of_7‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333134651184 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09358785301446915 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3800264298915863 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989609956741333 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014240025542676449 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26763877272605896 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8430213928222656 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5586956143379211 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2598194479942322 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 (particle/f_score‚0.35524243116378784 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2509259581565857 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2410714328289032 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4864406883716583 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27117592096328735 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34555816650390625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011532210744917393 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4413619637489319 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.33338314294815063 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4114350378513336 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.46759259700775146 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 jet/accuracy_1_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.40460124611854553 *jet/accuracy_2_of_6‚0.6410256028175354 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030113261193037033 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2832876443862915 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08436490595340729 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.24042147397994995 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.002116402145475149 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3304592967033386 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28108927607536316 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333432674408 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4732142388820648 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5544915199279785 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/0FRt2SRqqt1FBt/event_purity‚0.034591194242239 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.39857569336891174 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5940476655960083 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9875672459602356 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.295945942401886 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111109256744385 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4092622697353363 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.34125491976737976 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48211362957954407 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6088550686836243 -jet/accuracy_4_of_4‚0.0033374035265296698 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2855033576488495 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +particle/specificity‚0.9477376937866211 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.37162917852401733 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40330782532691956 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5716575384140015 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5366039872169495 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24871022999286652 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22325102984905243 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933383464813232 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.31618767976760864 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3385607600212097 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09306835383176804 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *loss/val_total_loss‚41.296452564583035 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1603952944278717 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.29520201683044434 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011771679855883121 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4102439880371094 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5718749761581421 particle/accuracy_3_of_3‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34861111640930176 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3817828893661499 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922829866409302 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4616679549217224 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34080812335014343 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9961451292037964 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5506765842437744 *jet/accuracy_1_of_5‚0.9275345206260681 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929848909378052 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2690136134624481 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0193258635699749 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_3_of_6‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007848469540476799 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,jet/accuracy_3_of_3‚0.025628257542848587 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30068227648735046 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 _step‚795 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5878060460090637 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1492070108652115 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10175987333059311 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21719059348106384 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.36772486567497253 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.32058820128440857 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2885526120662689 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2758531868457794 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3039351999759674 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181574344635 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.992449939250946 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6942441463470459 particle/accuracy_5_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3056650757789612 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19926318526268005 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.33932727575302124 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_5‚0.010618279688060284 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2840437591075897 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2976205348968506 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33839288353919983 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25933393836021423 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.30419331789016724 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2968409061431885 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.37166619300842285 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.33677494525909424 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0013227513991296291 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12980526685714722 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5821356773376465 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18805477023124695 particle/accuracy_2_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989529550075531 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012187737040221691 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3566780388355255 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23741181194782257 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.29379919171333313 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23914921283721924 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2680569291114807 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.37283962965011597 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019708994776010513 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3860282003879547 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9919624924659729 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.276737779378891 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.371232271194458 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 epoch‚7 % _timestamp‚1.7884377928247104e+09 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2738384008407593 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3149048388004303 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9850165843963623 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.25608327984809875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 particle/accuracy_5_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.40679824352264404 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5740740299224854 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.20455223321914673 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4863736629486084 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3488505482673645 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3271043598651886 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20972739160060883 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3592643141746521 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36732614040374756 +jet/accuracy_3_of_4‚0.05971681699156761 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2745990455150604 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 trainer/global_step‚9671 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6964063048362732 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6347448825836182 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5213386416435242 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4225592315196991 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.36063218116760254 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.30401235818862915 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03867621719837189 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15777777135372162 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5717739462852478 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10476189851760864 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028302613645792007 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23464739322662354 particle/accuracy_1_of_1‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10259301215410233 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.34970763325691223 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.32101893424987793 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8244149088859558 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.24088157713413239 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2899414300918579 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2273808866739273 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19460399448871613 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1622226983308792 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5842371582984924 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0k<ío! 474950410425663 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.38774552941322327 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936785697937012 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18805477023124695 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.30419331789016724 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.234375 jet/accuracy_6_of_7‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.31479284167289734 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3540540635585785 +particle/sensitivity‚0.2770829200744629 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.02483108639717102 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5466954708099365 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.28649958968162537 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21450619399547577 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3409067392349243 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21364851295948029 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3345775604248047 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3936026990413666 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15951290726661682 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.27718931436538696 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3923611044883728 particle/accuracy_2_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.37813788652420044 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5Ù¡„!ÑtÞ&¤¦ ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011771679855883121 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4102439880371094 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5718749761581421 particle/accuracy_3_of_3‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34861111640930176 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3817828893661499 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922829866409302 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4616679549217224 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34080812335014343 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9961451292037964 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5506765842437744 *jet/accuracy_1_of_5‚0.9275345206260681 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929848909378052 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2690136134624481 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0193258635699749 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_3_of_6‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007848469540476799 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,jet/accuracy_3_of_3‚0.025628257542848587 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30068227648735046 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 _step‚795 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5878060460090637 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1492070108652115 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10175987333059311 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21719059348106384 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.36772486567497253 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.32058820128440857 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2885526120662689 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2758531868457794 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3039351999759674 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181574344635 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.992449939250946 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6942441463470459 particle/accuracy_5_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3056650757789612 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19926318526268005 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.33932727575302124 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_5‚0.010618279688060284 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2840437591075897 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2976205348968506 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33839288353919983 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25933393836021423 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.30419331789016724 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2968409061431885 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.37166619300842285 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.33677494525909424 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0013227513991296291 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12980526685714722 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5821356773376465 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18805477023124695 particle/accuracy_2_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989529550075531 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012187737040221691 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3566780388355255 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23741181194782257 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.29379919171333313 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23914921283721924 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2680569291114807 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.37283962965011597 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019708994776010513 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3860282003879547 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9919624924659729 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.276737779378891 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.371232271194458 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 epoch‚7 % _timestamp‚1.7884377928247104e+09 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2738384008407593 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3149048388004303 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9850165843963623 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.25608327984809875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 particle/accuracy_5_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.40679824352264404 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5740740299224854 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.20455223321914673 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4863736629486084 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3488505482673645 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3271043598651886 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20972739160060883 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3592643141746521 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36732614040374756 +jet/accuracy_3_of_4‚0.05971681699156761 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2745990455150604 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 trainer/global_step‚9671 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6964063048362732 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6347448825836182 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5213386416435242 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4225592315196991 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.36063218116760254 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.30401235818862915 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03867621719837189 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15777777135372162 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5717739462852478 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10476189851760864 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028302613645792007 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23464739322662354 particle/accuracy_1_of_1‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10259301215410233 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.34970763325691223 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.32101893424987793 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8244149088859558 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.24088157713413239 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2899414300918579 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2273808866739273 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19460399448871613 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1622226983308792 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5842371582984924 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.38774552941322327 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936785697937012 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18805477023124695 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.30419331789016724 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.234375 jet/accuracy_6_of_7‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.31479284167289734 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3540540635585785 +particle/sensitivity‚0.2770829200744629 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.02483108639717102 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5466954708099365 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.28649958968162537 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21450619399547577 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3409067392349243 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21364851295948029 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3345775604248047 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3936026990413666 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15951290726661682 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.27718931436538696 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3923611044883728 particle/accuracy_2_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.37813788652420044 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38623514771461487 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27621692419052124 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 jet/accuracy_1_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.36911237239837646 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3739675283432007 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23533636331558228 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 _runtime‚9771.181930567 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2910611033439636 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.265398770570755 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 jet/accuracy_5_of_7‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5644841194152832 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019098583608865738 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24372969567775726 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 +jet/accuracy_2_of_2‚0.09975467622280121 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920675754547119 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5560247302055359 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_6‚0.1538461595773697 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.535765528678894 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18278688192367554 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2689165771007538 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03784959018230438 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14166148006916046 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36782121658325195 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8173867464065552 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_3‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.026757873594760895 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 6 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.344696968793869 particle/accuracy_4_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993274211883545 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.25613197684288025 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.37217116355895996 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.225862056016922 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_4‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0494663380086422 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.897315502166748 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 particle/accuracy_7_of_7‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333134651184 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09358785301446915 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3800264298915863 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989609956741333 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014240025542676449 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26763877272605896 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8430213928222656 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5586956143379211 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2598194479942322 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 (particle/f_score‚0.35524243116378784 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2509259581565857 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2410714328289032 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4864406883716583 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27117592096328735 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34555816650390625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.011532210744917393 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4413619637489319 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.33338314294815063 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4114350378513336 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.46759259700775146 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 jet/accuracy_1_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_p{”Zurity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.40460124611854553 *jet/accuracy_2_of_6‚0.6410256028175354 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030113261193037033 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2832876443862915 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08436490595340729 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.24042147397994995 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.002116402145475149 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3304592967033386 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28108927607536316 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333432674408 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4732142388820648 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5544915199279785 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/0FRt2SRqqt1FBt/event_purity‚0.034591194242239 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.39857569336891174 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5940476655960083 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9875672459602356 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.295945942401886 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111109256744385 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4092622697353363 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.34125491976737976 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48211362957954407 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6088550686836243 -jet/accuracy_4_of_4‚0.0033374035265296698 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2855033576488495 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +particle/specificity‚0.9477376937866211 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.37162917852401733 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40330782532691956 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5716575384140015 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5366039872169495 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24871022999286652 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22325102984905243 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933383464813232 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.31618767976760864 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3385607600212097 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09306835383176804 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *loss/val_total_loss‚41.296452564583035 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1603952944278717 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.29520201683044434 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835›öéÓ‘ß&‚(:† ©ÊåÔ¥¸Åµ gpu.0.gpu‚13.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚160.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÿr³zà&‚(:ï ©ÊåÔúŸ•· network.sent‚ 2.4549019e+07% disk./.usageGB‚286.66432189941406( disk.nvme0c0n1.out‚70472.24755859375 disk.dm-0.in‚ 466.90234375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.sda.out‚ 365.53515625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚70472.24755859375 disk.drbd2266.out‚0# disk.dm-0.out‚70472.24755859375 disk.sda2.in‚0 disk.rbd1.out‚ 5.212890625( proc.memory.availableMB‚ 334.23046875* proc.memory.percent‚18.019580841064453 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 365.53515625 network.recv‚ 3.284915e+06* disk./.usagePercent‚19.238283170056505 disk.rbd0.out‚0# memory_percent‚97.9600191116333! proc.memory.rssMB‚ 2952.328125 disk.drbd1716.out‚0 cpu‚25.29219488450614 disk.sda3.out‚ 365.53515625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.sda2.out‚0Mb4/žá&‚(:“ ¸ÊåÔÉÓ·µ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚149.651' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.930200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼³z{â&‚(:ð ¸ÊåÔæ倸! proc.memory.rssMB‚ 2952.328125 cpu‚25.291820551297462 network.sent‚ 2.4672468e+07 disk.sda3.out‚ 365.94140625 disk.rbd1.out‚ 5.212890625 network.recv‚ 3.290855e+06 disk.sda3.in‚0.203125 disk.drbd1716.out‚0& disk.nvme0n1.out‚70532.13720703125 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚70532.13720703125* disk./.usagePercent‚19.243612478896786 disk.dm-0.in‚ 466.90234375 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 466.90234375$ memory_percent‚97.95525074005127 disk.sda1.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚70532.13720703125 disk.drbd2265.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 466.90234375$ disk./.usageGB‚286.7437324523926 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.dm-1.out‚ 365.94140625* proc.memory.percent‚18.019580841064453 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda.out‚ 365.94140625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 335.01171875Üþ¹;2ã&- _wandbruntime‚9796  _runtime‚9796›„Çä&Á _runtime‚9796.793039398 _step‚796 & lr-AdamW/pg1‚2.342004308925363e-05 & lr-AdamW/pg2‚2.342004308925363e-05 trainer/global_step‚9699 % _timestamp‚1.7884378184380014e+09 n8|Ìå&Æ % _timestamp‚1.7884378184380014e+09 _runtime‚9796.793039398 _step‚796 & lr-AdamW/pg1‚2.342004308925363e-05 & lr-AdamW/pg2‚2.342004308925363e-05 trainer/global_step‚9699œ@gä2æ&- _wandbruntime‚9796  _runtime‚9796¥öø¤"(ç&œP ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1318715526557185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1341430267167232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320688806521856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20741487036928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383447155 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚9699 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1128383428382208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20744592430592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320688780130304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20709490702849 % _timestamp‚1.7884378184387708e+09 _runtime‚9796.794105751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104328737 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174315709 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20744592430592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20741487036928 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1128383456516608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174315703 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1298006010426368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320688780130304 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180647818 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1318712397660293 _step‚797 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1341430267167232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209108340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76319626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776ˆ@Z'(è&¡P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174315709 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20744592430592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20741487036928 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1128383456516608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174315703 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1298006010426368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320688780130304 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180647818 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1318712397660293 _step‚797 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1341430267167232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209108340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76319626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1318715526557185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1341430267167232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320688806521856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20741487036928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383447155 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚9699 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1128383428382208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20744592430592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320688780130304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20709490702849 % _timestamp‚1.7884378184387708e+09 _runtime‚9796.794105751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104328737 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302uÀÿM2é&- _wandbruntime‚9797  _runtime‚9797ˆå¾µê&¦N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20746704757760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20708496027025 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1341567005367808 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1320820300610048 OADeviceStatsMonito˸úw$r.on_train_batch_end/allocation.large_pool.freed‚ 174332987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209152562 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174332993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209152562 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104338990 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1298135432855552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104338993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1320820274218496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180665585 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20743599350784 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1318847032351709 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1341563873569280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383462417 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209129430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383462417 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76326595 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1298135407427584 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383485555 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1157258374625792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1318843903454609 _step‚798 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104338993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1320820274218496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180665585 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20743599350784 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1318847032351709 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1341563873569280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383462417 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209129430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383462417 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76326595 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1298135407427584 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383485555 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1157258374625792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1318843903454609 _step‚798 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28907336358400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28907285538304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20850222537216 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1348110601683456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210166192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210189298 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181546213 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚801 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9843.990332431 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76704823 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1327263510931456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1162911257126912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1348113733468672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1348110601683456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20847117143552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1327263484539904 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104841390 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104841387 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1325287700967301 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20847117143552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175179909 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210166192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76704313 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1134003999723008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1325290829864193 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20814943949057 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175179903 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20850222537216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1327263484539904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175179909 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175179903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096äJ/“&(ú& P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76704823 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1327263510931456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1162911257126912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1348113733468672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1348110601683456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20847117143552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1327263484539904 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104841390 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104841387 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1325287700967301 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20847117143552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175179909 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210166192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76704313 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1134003999723008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1325290829864193 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20814943949057 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175179903 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20850222537216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1327263484539904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175179909 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175179903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1162911336081408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28907336358400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28907285538304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20850222537216 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1348110601683456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210166192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210189298 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181546213 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚801 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9843.990332431 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6¡|÷˜ì2û&- _wandbruntime‚9844  _runtime‚9844âÛ§/+'ü&¥N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210187282 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385384469 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210187282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1327395005019648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175197187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325419206761617 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20849229457408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚802 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9844.881826298 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1163026703393792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1327394978628096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1163026622355456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175197193 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1348247339884032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1304605257488384 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20817052742365 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385384469 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20852334864384 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1134116410587136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210210414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385407607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104851643 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20813949273233 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1327394978628096 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175197187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76711959 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20849229457408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181564106 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1325422335658717 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181563599 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚9749 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1134116382452736úB %ý&ªN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210210414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385407607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104851643 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20813949273233 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1327394978628096 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175197187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76711959 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20849229457408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181564106 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1325422335658717 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181563599 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 trainer/global_step‚9749 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1134116382452736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210187282 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385384469 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210187282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1327395005019648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175197187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325419206761617 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20849229457408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚802 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9844.881826298 D;DeviceStatsMonitor.on_train_bat¨ Ä ch_end/segment.all.allocated‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1163026703393792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1327394978628096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1163026622355456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175197193 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1348247339884032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1304605257488384 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20817052742365 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385384469 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20852334864384 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1134116410587136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232¢àn332þ&- _wandbruntime‚9844  _runtime‚9844‘@w^ÿ&Ø 0loss/FBt2/assignment_loss‚0.3869203703258844 .loss/FBt1/detection_loss‚0.521149045947407 _step‚803 1loss/FBt4/assignment_loss‚0.07949395050213513 /loss/FRt1/detection_loss‚1.3351191324917109 % _timestamp‚1.7884378665280213e+09 /loss/FRt1/assignment_loss‚1.473012974387721 2loss/SRqqt4/assignment_loss‚1.3740268708080858 3loss/FBt1/assignment_loss‚0.0007592560238816556 /loss/FRt2/detection_loss‚2.4807802769872875 /loss/FRt3/detection_loss‚2.2782903924888496 1loss/SRqqt1/detection_loss‚0.8915394829608589 1loss/FBt3/assignment_loss‚0.23242549088899978 .loss/FBt3/detection_loss‚2.336573339998722 %loss/total_loss‚39.17152850987847 0loss/FRt2/assignment_loss‚3.5901674363348217 2loss/SRqqt1/assignment_loss‚0.8835015506764133 2loss/SRqqt2/assignment_loss‚1.6224368707626127 .loss/FRt4/detection_loss‚3.374016182290183 1loss/SRqqt4/detection_loss‚2.0380418743636155 /loss/FBt4/detection_loss‚1.9131349419292651 epoch‚8 2loss/SRqqt3/assignment_loss‚1.1968091689681604 trainer/global_step‚9749 0loss/SRqqt2/detection_loss‚2.242061615921557 /loss/FBt2/detection_loss‚1.9617610012783724 0loss/FRt3/assignment_loss‚2.4618402293750217 /loss/FRt4/assignment_loss‚2.818750520547231 1loss/SRqqt3/detection_loss‚1.6789165336196705 _runtime‚9844.882992058éQ¶c€'Ý /loss/FRt4/assignment_loss‚2.818750520547231 1loss/SRqqt3/detection_loss‚1.6789165336196705 _runtime‚9844.882992058 0loss/FBt2/assignment_loss‚0.3869203703258844 .loss/FBt1/detection_loss‚0.521149045947407 _step‚803 1loss/FBt4/assignment_loss‚0.07949395050213513 /loss/FRt1/detection_loss‚1.3351191324917109 % _timestamp‚1.7884378665280213e+09 /loss/FRt1/assignment_loss‚1.473012974387721 2loss/SRqqt4/assignment_loss‚1.3740268708080858 3loss/FBt1/assignment_loss‚0.0007592560238816556 /loss/FRt2/detection_loss‚2.4807802769872875 /loss/FRt3/detection_loss‚2.2782903924888496 1loss/SRqqt1/detection_loss‚0.8915394829608589 1loss/FBt3/assignment_loss‚0.23242549088899978 .loss/FBt3/detection_loss‚2.336573339998722 %loss/total_loss‚39.17152850987847 0loss/FRt2/assignment_loss‚3.5901674363348217 2loss/SRqqt1/assignment_loss‚0.8835015506764133 2loss/SRqqt2/assignment_loss‚1.6224368707626127 .loss/FRt4/detection_loss‚3.374016182290183 1loss/SRqqt4/detection_loss‚2.0380418743636155 /loss/FBt4/detection_loss‚1.9131349419292651 epoch‚8 2loss/SRqqt3/assignment_loss‚1.1968091689681604 trainer/global_step‚9749 0loss/SRqqt2/detection_loss‚2.242061615921557 /loss/FBt2/detection_loss‚1.9617610012783724 0loss/FRt3/assignment_loss‚2.4618402293750217£<0'‚(:’ ôÊåÔ¾þœµ gpu.0.gpu‚72.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚280.732' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.14640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Oã]v‚'‚(:ë ôÊåԼη disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 466.90234375% disk.nvme0n1.out‚70789.2236328125* disk./.usagePercent‚19.248829400429535 disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚70789.2236328125 disk.sda3.out‚ 368.37109375 disk.drbd1716.in‚0 disk.sda2.out‚0 cpu‚25.29032938796499 network.sent‚ 2.4798975e+07 network.recv‚ 3.310035e+06 disk.rbd1.out‚ 5.212890625" disk.nvme0c0n1.in‚ 466.90234375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 368.37109375$ memory_percent‚97.96452522277832' disk.nvme0c0n1.out‚70789.2236328125 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.rbd0.out‚0! proc.memory.rssMB‚ 2952.328125 proc.cpu.threads‚17$ disk./.usageGB‚286.8214683532715 disk.drbd2080.out‚0' proc.memory.availableMB‚ 333.4921875* proc.memory.percent‚18.019580841064453 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 368.37109375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0F”²Ö’ƒ'‚(:‡ ƒËåÔ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚286.885' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.377 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷«çx„'‚(:í ƒËåÔç“÷¶' proc.memory.availableMB‚ 332.9921875! proc.memory.rssMB‚ 2952.328125 disk.nvme0n1.in‚ 466.90234375 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125* proc.memory.percent‚18.019580841064453 network.sent‚ 2.4907967e+07 disk.rbd0.out‚0 disk.rbd1.out‚ 5.212890625 network.recv‚ 3.316974e+06% disk./.usageGB‚286.84064865112305& disk.nvme0n1.out‚70835.55615234375 disk.sda2.out‚0$ memory_percent‚97.96757698059082 cpu‚25.289942899631427* disk./.usagePercent‚19.250116606311696 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 466.90234375( disk.nvme0c0n1.out‚70835.55615234375 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 369.0078125 disk.drbd2266.out‚0 disk.dm-0.in‚ 466.90234375# disk.dm-0.out‚70835.55615234375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 369.0078125 disk.sda1.in‚0 disk.sda.out‚ 369.0078125 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd1716.out‚0†ac“…'‚(:ˆ ’ËåÔðÔ›µ gpu.0.gpu‚85.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚289.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.9778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ Ru†'‚(:ê ’ËåÔޡ̶ network.sent‚ 2.4912924e+07 disk.dm-1.in‚0.203125" disk.dm-0.out‚70889.1767578125 disk.sda.out‚ 369.57421875* proc.memory.percent‚18.019580841064453 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 333.48046875$ disk./.usageGB‚286.8587532043457 disk.rbd1.in‚0 disk.sda3.out‚ 369.57421875 disk.sda1.out‚0$ memory_percent‚97.96459674835205 disk.drbd2266.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚70889.1767578125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0( disk./.usagePercent‚19.2513316180692 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚70889.1767578125 proc.cpu.threads‚17 network.recv‚ 3.320413e+06 disk.dm-0.in‚ 466.90234375 disk.sda1.in‚0 disk.sda.in‚0.203125! proc.memory.rssMB‚ 2952.328125 cpu‚25.28958153132843 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 369.57421875 disk.rbd1.out‚ 5.212890625" disk.nvme0c0n1.in‚ 466.90234375 disk.nvme0n1.in‚ 466.90234375N 2‡'- _wandbruntime‚9891  _runtime‚9891”¦\Ɉ'à _runtime‚9891.108517681 _step‚804 ' lr-AdamW/pg1‚2.1572581620206633e-05 ' lr-AdamW/pg2‚2.1572581620206633e-05 trainer/global_step‚9799 % _timestamp‚1.7884379127534559e+09R~¾#Ή'È ' lr-AdamW/pg2‚2.1572581620206633e-05 trainer/global_step‚9799 % _timestamp‚1.7884379127534559e+09 _runtime‚9891.108517681 _step‚804 ' lr-AdamW/pg1‚2.1572581620206633e-05¤Ìù3e2Š'- _wandbruntime‚9891  _runtime‚9891åP†(‹'™P UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1333838215341056 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1354790936199680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211224044 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884379127542143e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176044103 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20955852643840 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29055017554944 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1331866133171201 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1354794067984896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211247150 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1139624514795008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29055070472192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176044109 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77088146 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1333838188949504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1310945735975936 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105354037 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387268147 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1331863004274309 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1354790936199680 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176044109 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176044109 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77088146 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1333838188949504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1310945735975936 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105354037 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387268147 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1331863004274309 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1354790936199680 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176044109 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1354790936199680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211224044 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884379127542143e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176044103 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20955852643840 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29055017554944 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1331866133171201 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1354794067984896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211247150 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664¥78Y2'- _wandbruntime‚9891  _runtime‚98914z¾àŽ'¦N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1333969683037696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1354924542601728 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1354927674400256 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20954859564032 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105364293 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20919402519441 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1331997638965725 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387329659 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176061393 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20957964971008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176061387 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1168794880522240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77095599 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1139736925659136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182459889 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20922505988573 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211268266 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77096089 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884379136433597e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29057954863104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1139736953793536 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 trainer/global_step‚9799 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176061387 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ä·â1''«N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1331997638965725 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387329659 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176061393 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20957964971008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176061387 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1168794880522240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77095599 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1139736925659136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182459889 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20922505988573 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211268266 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77096089 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 % _timestamp‚1.7884379136433597e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29057954863104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1139736953793536 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 trainer/global_step‚9799 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176061387 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1333969683037696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1354924542601728 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1354927674400256 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20954859564032 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105364293 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20919402519441 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302¦¸ ¯2'- _wandbruntime‚9892  _runtime‚98921£úÕ\‘'Ö 0loss/FBt4/assignment_loss‚0.2785638231193604 1loss/SRqqt1/detection_loss‚1.0352732979609995 _step‚807 1loss/FBt2/assignment_loss‚0.22763120802119374 /loss/FRt2/detection_loss‚2.6105633030335107 2loss/SRqqt3/assignment_loss‚0.8386807933077216 3loss/FBt1/assignment_loss‚0.0004148334334313999 /loss/FBt4/detection_loss‚2.1003339792552747 epoch‚8 /loss/FRt2/assignment_loss‚3.376608353522089 0loss/SRqqt4/assignment_loss‚1.40686037820397 1loss/SRqqt4/detection_loss‚2.1664426471247817 0loss/FRt3/assignment_loss‚2.6134155486759387 /loss/FBt3/assignment_loss‚0.141142545582746 /loss/FRt4/detection_loss‚3.3118301413275977 0loss/SRqqt2/detection_loss‚2.579462596864411 % _timestamp‚1.7884379136460752e+09 2loss/SRqqt1/assignment_loss‚0.9594458893554497 2loss/SRqqt2/assignment_loss‚1.5875430826210615 .loss/FRt3/detection_loss‚2.519379420500053 /loss/FBt2/detection_loss‚2.7195850781032016 /loss/FBt1/detection_loss‚0.7091687985488938 .loss/FBt3/detection_loss‚2.071943259239197 %loss/total_loss‚40.61716356068821 trainer/global_step‚9799 _runtime‚9892.001057519 /loss/FRt1/detection_loss‚1.5174417942762375 0loss/FRt1/assignment_loss‚1.3898008344829946 0loss/FRt4/assignment_loss‚2.8039964789694007 1loss/SRqqt3/detection_loss‚1.6516354751586915VdÞa’'Û /loss/FRt2/detection_loss‚2.6105633030335107 2loss/SRqqt3/assignment_loss‚0.8386807933077216 3loss/FBt1/assignment_loss‚0.0004148334334313999 /loss/FBt4/detection_loss‚2.1003339792552747 epoch‚8 /loss/FRt2/assignment_loss‚3.376608353522089 0loss/SRqqt4/assignment_loss‚1.40686037820397 1loss/SRqqt4/detection_loss‚2.1664426471247817 0loss/FRt3/assignment_loss‚2.6134155486759387 /loss/FBt3/assignment_loss‚0.141142545582746 /loss/FRt4/detection_loss‚3.3118301413275977 0loss/SRqqt2/detection_loss‚2.579462596864411 % _timestamp‚1.7884379136460752e+09 2loss/SRqqt1/assignment_loss‚0.9594458893554497 2loss/SRqqt2/assignment_loss‚1.5875430826210615 .loss/FRt3/detection_loss‚2.519379420500053 /loss/FBt2/detection_loss‚2.7195850781032016 /loss/FBt1/detection_loss‚0.7091687985488938 .loss/FBt3/detection_loss‚2.071943259239197 %loss/total_loss‚40.61716356068821 trainer/global_step‚9799 _runtime‚9892.001057519 /loss/FRt1/detection_loss‚1.5174417942762375 0loss/FRt1/assignment_loss‚1.3898008344829946 0loss/FRt4/assignment_loss‚2.8039964789694007 1loss/SRqqt3/detection_loss‚1.6516354751586915 0loss/FBt4/assignment_loss‚0.2785638231193604 1loss/SRqqt1/detection_loss‚1.0352732979609995 _step‚807 1loss/FBt2/assignment_loss‚0.22763120802119374§†×+Þ’“'‚(:‡ ¡ËåÔ¡¸Æµ gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚298.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.7116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â:o.i”'‚(:Þ ¡ËåÔ°¬Â¸ network.sent‚ 2.4919269e+07% disk.nvme0n1.out‚70919.9736328125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 333.2109375* proc.memory.percent‚18.019580841064453 disk.sda2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚70919.9736328125 disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚70919.9736328125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 466.90234375 proc.cpu.threads‚17 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0* disk./.usagePercent‚19.252339007716436% disk./.usageGB‚286.87376403808594 disk.rbd1.out‚ 5.212890625 disk.rbd0.out‚0 network.recv‚ 3.325047e+06 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda3.out‚370.125 disk.drbd2265.in‚0$ memory_percent‚97.96624183654785 cpu‚25.289209195843135 disk.dm-1.out‚370.125 disk.drbd2266.out‚0 disk.sda.out‚370.125! proc.memory.rssMB‚ 2952.328125ëŸÁ“•'‚(:ˆ °ËåÔ¥Ÿµ gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚188.661' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.7322 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á•Ýp–'‚(:å °ËåÔ–Çð¶ network.recv‚ 3.330903e+06 disk.drbd1716.out‚0 disk.sda.out‚ 370.6484375 disk.rbd0.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2952.328125 disk.sda2.in‚0 network.sent‚ 2.5027711e+07 disk.sda3.out‚ 370.6484375 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 466.90234375) disk./.usagePercent‚19.25390833390133 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 proc.cpu.threads‚17$ disk./.usageGB‚286.8971481323242% disk.nvme0n1.out‚70970.5673828125 disk.drbd2266.out‚0$ memory_percent‚97.96686172485352* proc.memory.percent‚18.019580841064453 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda.in‚0.203125 disk.sda3.in‚0.203125 disk.sda1.out‚0& proc.memory.availableMB‚ 333.109375 cpu‚25.2888569794592 disk.drbd2080.out‚0 disk.dm-1.out‚ 370.6484375 disk.rbd1.out‚ 5.767578125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 466.90234375' disk.nvme0c0n1.out‚70970.5673828125 disk.dm-0.in‚ 466.90234375" disk.dm-0.out‚70970.5673828125(n'‘—'‚(:† ¿ËåÔï“ÿ´ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚302.927' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.5854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Ù¼¨x˜'‚(:í ¿ËåÔŽ¾é·" disk.nvme0c0n1.in‚ 471.90234375( proc.memory.availableMB‚ 332.86328125 disk.drbd1716.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 371.47265625 disk.dm-0.in‚ 471.90234375 network.sent‚ 2.5033761e+07 network.recv‚ 3.335283e+06 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0$ memory_percent‚97.96836376190186* proc.memory.percent‚18.019580841064453 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚19.253666406781473 disk.drbd2080.in‚0 disk.drbd2265.out‚0" disk.dm-0.out‚71020.5556640625 cpu‚25.288517294425183$ disk./.usageGB‚286.8935432434082 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚71020.5556640625 disk.sda2.in‚0 disk.rbd1.out‚ 5.830078125' disk.nvme0c0n1.out‚71020.5556640625 disk.drbd2265.in‚0 disk.sda3.out‚ 371.47265625! proc.memory.rssMB‚ 2952.328125 disk.drbd1716.out‚0 disk.sda.out‚ 371.47265625 disk.sda2.out‚0ãéF2™'- _wandbruntime‚9938  _runtime‚9938úÂ~‹Æš'À & lr-AdamW/pg1‚2.067398608723091e-05 & lr-AdamW/pg2‚2.067398608723091e-05 trainer/global_step‚9849 % _timestamp‚1.7884379596939797e+09 _runtime‚ 9938.04902116 _step‚808½–§Ë›'Å & lr-AdamW/pg1‚2.067398608723091e-05 & lr-AdamW/pg2‚2.067398608723091e-05 trainer/global_step‚9849 % _timestamp‚1.7884379596939797e+09 _runtime‚ 9938.04902116 _step‚808¨píÄ2œ'- _wandbruntime‚9938  _runtime‚9938‚ðze('œP G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1145245086135808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1361471270715904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176908303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21061482750464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389213311 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212305002 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1145245058001408 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1340412919750656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21058377356800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105866690 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1174447791393792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389190199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1361471270715904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1174447712439296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884379596947417e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1340412919750656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77474295 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984tK'(ž'¡P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1317415586036736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21022746972549 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77474787 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1317415560608768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212305002 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1361474402501120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183341477 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338441436478209 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212281896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1174447791393792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389190199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1361471270715904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1174447712439296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884379596947417e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1340412919750656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77474295 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1145245086135808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1361471270715904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176908303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21061482750464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389213311 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212305002 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1145245058001408 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1340412919750656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21058377356800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105866690 JÍ&,' '¦N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176925593 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338569813375633 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183359300 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389251711 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1340544387447296 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212326118 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212302986 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21063595077632 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1174563089647104 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚810 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389228573 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29205671588352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1317544957609984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176925587 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21024855765649 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1338572942272733 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105876940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1317544983037952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1361604877117952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 _runtime‚9938.943696683 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212302986 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1361608008916480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183358795 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1145357496999936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176925593 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1145357468865536 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29205620781568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176925587 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389251711 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 % _timestamp‚1.7884379605880802e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212326118 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚04\õ^ ¡'«N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77481855 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1174563168588288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21063595077632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1361608008916480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183358795 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1145357496999936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176925593 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1145357468865536 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29205620781568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176925587 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389251711 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 % _timestamp‚1.7884379605880802e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212326118 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176925593 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338569813375633 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183359300 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389251711 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1340544387447296 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212326118 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212302986 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21063595077632 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1174563089647104 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _step‚810 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389228573 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29205671588352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1317544957609984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176925587 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21024855765649 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_”5ôÎÓbytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1338572942272733 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105876940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1317544983037952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1361604877117952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 _runtime‚9938.943696683 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212302986 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1368151605232128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1180215822529024 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106379337 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21131303687681 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1368154737017344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77856499 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1368151605232128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1368151605232128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1180215822529024 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106379337 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21131303687681 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1368154737017344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77856499 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1368151605232128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177789793 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1347119091856896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177789787 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚9899 B8Devi@BfŠõceStatsMonitor.on_train_batch_end/active.all.current‚23138 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1180331167003648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213383970 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391173763 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1345145116682641 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1347119118248448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _runtime‚9986.174367457 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21169225184256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391150625 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213383970 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1150978040206336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21130309011857 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21133412480989 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1368288343432704 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1368288343432704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184253476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29353154931712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1324014833098752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21166119777280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29353207835648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391173763 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1347119091856896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177789793 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1150978012071936 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21166119777280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77863886 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213360838 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1345148245579741 % _timestamp‚1.7884380078184779e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1368285211634176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77864405 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 trainer/global_step‚9899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1180331167003648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213383970 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391173763 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1345145116682641 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1347119118248448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _runtime‚9986.174367457 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21169225184256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391150625 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213383970 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1150978040206336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21130309011857 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21133412480989 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1368288343432704 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1368288343432704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184253476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29353154931712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1324014833098752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21166119777280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29353207835648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391173763 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1347119091856896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177789793 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1150978012071936 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21166119777280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77863886 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213360838 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1345148245579741 % _timestamp‚1.7884380078184779e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1368285211634176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77864405 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0®ä—\Ì2´'- _wandbruntime‚9986  _runtime‚9986isÃUµ'Ï 0loss/FRt3/assignment_loss‚2.7252104327082636 /loss/FRt4/assignment_loss‚3.945974356853045 2loss/FBt1/assignment_loss‚0.002717891601538014 /loss/FRt4/detection_loss‚2.9578806368204265 .loss/FBt4/detection_loss‚2.099828541278839 $ _timestamp‚1.788438007820865e+09 1loss/SRqqt1/assignment_loss‚1.018460915322943 1loss/SRqqt3/assignment_loss‚1.101782013938107 /loss/FBt2/assignment_loss‚0.455524713753468 trainer/global_step‚9899 _runtime‚9986.175877768 2loss/SRqqt2/assignment_loss‚1.5508506149053574 -loss/FRt2/detection_loss‚2.54008030540803 1loss/SRqqt1/detection_loss‚0.8035887521074478 0loss/FRt1/assignment_loss‚1.4292071769634882 1loss/SRqqt4/assignment_loss‚1.619901062833378 _step‚815 epoch‚8 0loss/FBt4/assignment_loss‚0.1903103015004035 /loss/FBt1/detection_loss‚0.6978084714048438 .loss/FBt2/detection_loss‚2.010894975878976 /loss/FBt3/detection_loss‚2.5392713993787766 0loss/FRt2/assignment_loss‚3.5284775109852062 1loss/FBt3/assignment_loss‚0.00662770752872414 /loss/FRt1/detection_loss‚1.6560536330876252 /loss/FRt3/detection_loss‚2.6559546917676924 1loss/SRqqt3/detection_loss‚1.6597932560576334 $loss/total_loss‚41.7360428738519 1loss/SRqqt2/detection_loss‚2.2853213250637054 /loss/SRqqt4/detection_loss‚2.25452218670398=ðêOZ¶'Ô 0loss/FRt2/assignment_loss‚3.5284775109852062 1loss/FBt3/assignment_loss‚0.00662770752872414 /loss/FRt1/detection_loss‚1.6560536330876252 /loss/FRt3/detection_loss‚2.6559546917676924 1loss/SRqqt3/detection_loss‚1.6597932560576334 $loss/total_loss‚41.7360428738519 1loss/SRqqt2/detection_loss‚2.2853213250637054 /loss/SRqqt4/detection_loss‚2.25452218670398 0loss/FRt3/assignment_loss‚2.7252104327082636 /loss/FRt4/assignment_loss‚3.945974356853045 2loss/FBt1/assignment_loss‚0.002717891601538014 /loss/FRt4/detection_loss‚2.9578806368204265 .loss/FBt4/detection_loss‚2.099828541278839 $ _timestamp‚1.788438007820865e+09 1loss/SRqqt1/assignment_loss‚1.018460915322943 1loss/SRqqt3/assignment_loss‚1.101782013938107 /loss/FBt2/assignment_loss‚0.455524713753468 trainer/global_step‚9899 _runtime‚9986.175877768 2loss/SRqqt2/assignment_loss‚1.5508506149053574 -loss/FRt2/detection_loss‚2.54008030540803 1loss/SRqqt1/detection_loss‚0.8035887521074478 0loss/FRt1/assignment_loss‚1.4292071769634882 1loss/SRqqt4/assignment_loss‚1.619901062833378 _step‚815 epoch‚8 0loss/FBt4/assignment_loss‚0.1903103015004035 /loss/FBt1/detection_loss‚0.6978084714048438 .loss/FBt2/detection_loss‚2.010894975878976 /loss/FBt3/detection_loss‚2.5392713993787766¯Ä}JA“·'‚(:ˆ ûËåÔùñ¦¶ gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚44.0 gpu.0.powerWatts‚126.078' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Áƒàr¸'‚(:ç ûËåÔúø²¸ disk.sda2.in‚0 disk.rbd0.in‚0 cpu‚25.287060369561193 disk.sda2.out‚0 disk.drbd2265.out‚0$ memory_percent‚97.96748161315918' proc.memory.availableMB‚ 333.0078125* proc.memory.percent‚18.019580841064453 disk.drbd2266.out‚0 disk.rbd1.out‚ 6.5498046875 network.sent‚ 2.5160186e+07( disk.nvme0c0n1.out‚71217.38427734375# disk.dm-0.out‚71217.38427734375 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 373.671875 network.recv‚ 3.35439e+06( disk./.usagePercent‚19.2559328414821 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.in‚0.203125& disk.nvme0n1.out‚71217.38427734375 disk.drbd2080.out‚0 disk.dm-1.out‚ 373.671875" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375! proc.memory.rssMB‚ 2952.328125 disk.rbd1.in‚0$ disk./.usageGB‚286.9273147583008 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.sda.in‚0.203125 disk.sda3.out‚ 373.671875 disk.drbd2080.in‚0³’ɶž¹'‚(:“ ŠÌåÔ»¸ð´ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚152.676' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.535199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·DÏadº'‚(:Ù ŠÌåÔ‹„¸$ disk./.usageGB‚286.9376411437988 disk.drbd1716.out‚0 network.sent‚ 2.5268906e+07 disk.sda.out‚ 374.03125" disk.dm-0.out‚71257.1220703125 disk.sda3.out‚ 374.03125 disk.sda1.in‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2952.328125 cpu‚25.28668292970968 disk.sda.in‚0.203125 disk.rbd0.out‚0% disk.nvme0n1.out‚71257.1220703125 disk.sda2.in‚0 disk.rbd1.out‚ 6.5498046875' disk.nvme0c0n1.out‚71257.1220703125 disk.sda1.out‚0* proc.memory.percent‚18.019580841064453 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda2.out‚0 disk.dm-1.in‚0.203125" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.recv‚ 3.360333e+06 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0* disk./.usagePercent‚19.256625853877292 disk.rbd0.in‚0 disk.dm-1.out‚ 374.03125" memory_percent‚97.967529296875 proc.memory.availableMB‚333 disk.nvme0n1.in‚ 471.90234375 disk.rbd1.in‚0z=ÝRž»'‚(:“ ™ÌåÔ·¬—µ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚132.573' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.514599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢êêq¼'‚(:æ ™ÌåÔŸÁû¶ disk.sda.in‚0.203125$ memory_percent‚97.96905517578125) disk./.usagePercent‚19.25776867151014 disk.rbd0.in‚0 disk.rbd1.out‚ 6.6123046875 disk.sda3.out‚ 374.60546875 disk.sda.out‚ 374.60546875 disk.rbd0.out‚0 disk.dm-1.out‚ 374.60546875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 network.sent‚ 2.5275205e+07 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.in‚0" proc.memory.availableMB‚332.75! proc.memory.rssMB‚ 2952.328125 disk.sda2.in‚0 disk.dm-1.in‚0.203125" disk.dm-0.out‚71296.7548828125 disk.drbd2080.out‚0 network.recv‚ 3.365036e+06% disk.nvme0n1.out‚71296.7548828125 disk.sda1.out‚0 disk.drbd2080.in‚0 cpu‚25.28633509293191 proc.cpu.threads‚17$ disk./.usageGB‚286.9546699523926 disk.drbd2265.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚71296.7548828125* proc.memory.percent‚18.019580841064453 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0&D)Í4½'/ _wandbruntime‚10032  _runtime‚100326=‡¤É¾'à trainer/global_step‚9949 $ _timestamp‚1.788438053948383e+09 _runtime‚10032.303496447 _step‚816 ' lr-AdamW/pg1‚1.8928001715160226e-05 ' lr-AdamW/pg2‚1.8928001715160226e-05½išÎ¿'È $ _timestamp‚1.788438053948383e+09 _runtime‚10032.303496447 _step‚816 ' lr-AdamW/pg1‚1.8928001715160226e-05 ' lr-AdamW/pg2‚1.8928001715160226e-05 trainer/global_step‚9949°Gó€4À'/ _wandbruntime‚10032  _runtime‚10032Â?jtÁ'P TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178636703 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106891990 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106891987 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1353562328569856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21236756933889 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1330355260730368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1185983127784448 trainer/global_step‚9949 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1374835071533568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214420706 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29496983370240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚566 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1374831939748352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178636709 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185131085 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10032.304393009 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178636703 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393057415 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1353562328569856 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1351592043092225 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21269637570048 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1353562302178304 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1353562302178304 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185131651 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21272742963712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21272742963712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1374835071533568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214420706 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1156486144414208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚ÍNXL¯ 27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1374831939748352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214397600 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21233653464965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1156486172548608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚817 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78239098 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1185983210933248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 % _timestamp‚1.7884380539491322e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21233653464965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1156486172548608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚817 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78239098 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1185983210933248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 % _timestamp‚1.7884380539491322e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106891987 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1353562328569856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21236756933889 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1330355260730368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1185983127784448 trainer/global_step‚9949 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1374835071533568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214420706 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29496983370240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚566 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1374831939748352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178636709 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185131085 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10032.304393009 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178636703 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393057415 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1353562328569856 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1351592043092225 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21269637570048 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1353562302178304 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1353562302178304 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185131651 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21272742963712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21272742963712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1374835071533568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214420706 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1156486144414208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1374831939748352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302±»Roz4Ã'/ _wandbruntime‚10033  _runtime‚10033Y‰{ò-'Ä'§N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 178653987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1156598555278336 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1374968677948928 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1353693796266496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214441822 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚9949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1156598583412736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185148789 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21271749883904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393095815 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106902243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21238865727197 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 178653993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178653987 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1353693822658048 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1353693796266496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78246549 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185149317 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214441822 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1353693822658048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106902240 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚818 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976¸ã ¥#Å'¬N EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1374968677948928 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1353693796266496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214441822 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 trainer/global_step‚9949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1156598583412736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185148789 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21271749883904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393095815 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106902243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21238865727197 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 178653993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178653987 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1353693822658048 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1353693796266496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78246549 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185149317 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214441822 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1353693822658048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106902240 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚818 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 178653987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1156598555278336 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78621106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179500909 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1381515406049792 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1360137032979456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179500903 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107404640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29644365952000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394979467 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1191751000655360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394956355 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179500903 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21375267676672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1336825110791168 trainer/global_step‚9999 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215478558 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215455452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1162106687620608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚821 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1360137006587904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179500909 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21339106711173 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381512274264576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1381515406049792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21378373070336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107404637 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1162106715755008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788438100937047e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1360137006587904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381512274264576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1381515406049792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21378373070336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107404637 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1162106715755008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788438100937047e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1360137006587904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78621106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179500909 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1381515406049792 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1360137032979456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179500903 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107404640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29644365952000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394979467 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1191751000655360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394956355 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179500903 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21375267676672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1336825110791168 trainer/global_step‚9999 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215478558 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215455452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1162106687620608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _step‚821 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1360137006587904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179500909 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21339106711173µ=å­“Õ'‚(:ˆ ÕÌåÔþœŸµ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.912' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­§çªvÖ'‚(:ë ÕÌåÔë½· network.recv‚ 3.384515e+06" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚71497.3466796875 disk.sda3.in‚0.203125 disk.sda3.out‚ 376.85546875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 6.80078125$ disk./.usageGB‚286.9039001464844 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0$ memory_percent‚97.96786308288574 proc.cpu.threads‚17 disk.dm-1.out‚ 376.85546875 disk.rbd1.in‚0 disk.sda.out‚ 376.85546875 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0* proc.memory.percent‚18.019580841064453 disk.drbd2080.out‚0 disk.sda.in‚0.203125 cpu‚25.284879783490275 network.sent‚ 2.5401847e+07 disk.dm-0.in‚ 471.90234375 disk.dm-1.in‚0.203125" disk.dm-0.out‚71497.3466796875' proc.memory.availableMB‚ 332.9453125* disk./.usagePercent‚19.254361467236933% disk.nvme0n1.out‚71497.3466796875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2952.328125\Zp4×'/ _wandbruntime‚10080  _runtime‚10080&ˆß\VØ'§N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚822 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21377379990528 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107414890 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21341215504273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215476542 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _runtime‚10080.178010225 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1381649012465152 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1191866355613184 ;4DeviceStatsMonitor.on_train_batch_eÅ| × nd/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78628722 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1381645880666624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21380485397504 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107414893 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215499674 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1360268527067648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1360268527067648 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179518193 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29647310032384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29647257128448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179518187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179518187 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215476542 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186043615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395017867 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78628202 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179518193 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21377379990528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395017867 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186043092 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21344318973405 trainer/global_step‚9999 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884381018223228e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1191866436651520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21380485397504ð„æ¤2'Ù'¬N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21344318973405 trainer/global_step‚9999 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884381018223228e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1191866436651520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21380485397504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚822 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21377379990528 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107414890 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21341215504273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215476542 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 _runtime‚10080.178010225 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1381649012465152 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1191866355613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78628722 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1381645880666624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21380485397504 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107414893 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215499674 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1360268527067648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1360268527067648 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179518193 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29647310032384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29647257128448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179518187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179518187 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215476542 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186043615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395017867 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78628202 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179518193 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21377379990528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395017867 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186043092 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552¶Ü¢?Ã4Ú'/ _wandbruntime‚10080  _runtime‚10080£Ñ)`Û'Ú /loss/FRt2/assignment_loss‚3.643884505981054 0loss/FBt2/assignment_loss‚0.4992092944739852 1loss/FBt4/assignment_loss‚0.29891138036096393 1loss/SRqqt1/detection_loss‚0.8842918501399896 /loss/FBt1/detection_loss‚0.7278923221637268 /loss/FBt3/detection_loss‚2.3486998826265335 $loss/total_loss‚41.8796334417164 epoch‚8 2loss/SRqqt2/assignment_loss‚1.5008755640993858 2loss/SRqqt3/assignment_loss‚1.1700757995925166 /loss/FRt2/detection_loss‚2.3989950235073385 /loss/FRt4/assignment_loss‚3.333943856389899 1loss/SRqqt4/assignment_loss‚1.380585557092791 /loss/FRt4/detection_loss‚3.4444902319657174 1loss/SRqqt4/detection_loss‚2.5575258343116096 .loss/FBt2/detection_loss‚2.130580951770147 0loss/FRt1/assignment_loss‚1.5287421382963657 0loss/FRt3/assignment_loss‚2.6800690972805024 0loss/SRqqt2/detection_loss‚2.458892828431623 /loss/FBt4/detection_loss‚2.3626840141686527 trainer/global_step‚9999 _runtime‚10080.179066649 /loss/FRt3/detection_loss‚2.2622266495227814 3loss/FBt1/assignment_loss‚0.0009306705777747213 /loss/FRt1/detection_loss‚1.4326799197588116 1loss/SRqqt3/detection_loss‚1.8095903125676243 2loss/SRqqt1/assignment_loss‚1.0037094061016378 2loss/FBt3/assignment_loss‚0.020146350534968605 % _timestamp‚1.7884381018241048e+09 _step‚823ÔÙ]ZeÜ'ß /loss/FRt1/detection_loss‚1.4326799197588116 1loss/SRqqt3/detection_loss‚1.8095903125676243 2loss/SRqqt1/assignment_loss‚1.0037094061016378 2loss/FBt3/assignment_loss‚0.020146350534968605 % _timestamp‚1.7884381018241048e+09 _step‚823 /loss/FRt2/assignment_loss‚3.643884505981054 0loss/FBt2/assignment_loss‚0.4992092944739852 1loss/FBt4/assignment_loss‚0.29891138036096393 1loss/SRqqt1/detection_loss‚0.8842918501399896 /loss/FBt1/detection_loss‚0.7278923221637268 /loss/FBt3/detection_loss‚2.3486998826265335 $loss/total_loss‚41.8796334417164 epoch‚8 2loss/SRqqt2/assignment_loss‚1.5008755640993858 2loss/SRqqt3/assignment_loss‚1.1700757995925166 /loss/FRt2/detection_loss‚2.3989950235073385 /loss/FRt4/assignment_loss‚3.333943856389899 1loss/SRqqt4/assignment_loss‚1.380585557092791 /loss/FRt4/detection_loss‚3.4444902319657174 1loss/SRqqt4/detection_loss‚2.5575258343116096 .loss/FBt2/detection_loss‚2.130580951770147 0loss/FRt1/assignment_loss‚1.5287421382963657 0loss/FRt3/assignment_loss‚2.6800690972805024 0loss/SRqqt2/detection_loss‚2.458892828431623 /loss/FBt4/detection_loss‚2.3626840141686527 trainer/global_step‚9999 _runtime‚10080.179066649 /loss/FRt3/detection_loss‚2.2622266495227814 3loss/FBt1/assignment_loss‚0.0009306705777747213·i„åì“Ý'‚(:ˆ äÌåÔů¯¶ gpu.0.gpu‚85.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚283.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“¿âpÞ'‚(:å äÌåÔÁÙª·% disk./.usageGB‚286.93779373168945 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375% disk.nvme0n1.out‚71543.4365234375 disk.drbd2266.out‚0 network.recv‚ 3.389956e+06 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 6.80078125 disk.drbd1716.out‚0) disk./.usagePercent‚19.25663609417866 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.out‚ 377.296875 disk.drbd2080.in‚0$ memory_percent‚97.96812534332275 disk.drbd1716.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda.out‚ 377.296875( proc.memory.availableMB‚ 332.90234375* proc.memory.percent‚18.019580841064453 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 377.296875' disk.nvme0c0n1.out‚71543.4365234375 disk.drbd2080.out‚0 cpu‚25.28448296870739" disk.dm-0.out‚71543.4365234375 disk.sda3.in‚0.203125 disk.sda.in‚0.203125! proc.memory.rssMB‚ 2952.328125 proc.cpu.threads‚17 network.sent‚ 2.5498512e+07äÇ Wžß'‚(:“ óÌåÔßηµ gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.032' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.206399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kãAvà'‚(:ë óÌåÔµ‡ë¶ disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.out‚ 6.95703125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 377.99609375 disk.rbd1.in‚0 disk.drbd2080.out‚0 network.recv‚ 3.39646e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 471.90234375* disk./.usagePercent‚19.257678556858085 disk.drbd1716.in‚0$ memory_percent‚97.96817302703857 network.sent‚ 2.5596839e+07 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚71576.2646484375 disk.sda1.in‚0( proc.memory.availableMB‚ 332.89453125! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453$ disk./.usageGB‚286.9533271789551 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda3.out‚ 377.99609375 disk.sda.in‚0.203125 disk.drbd2265.out‚0% disk.nvme0n1.out‚71576.2646484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0" disk.dm-0.out‚71576.2646484375 disk.drbd1716.out‚0 disk.sda.out‚ 377.99609375 disk.drbd2080.in‚0 cpu‚25.284138553817055éIl?á'‚(:’ ‚ÍåÔý‡“µ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚259.954' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.99080000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R Ÿvvâ'‚(:ë ‚ÍåÔ¸‘¶¶ disk.nvme0n1.in‚ 471.90234375& proc.memory.availableMB‚ 332.890625 network.recv‚ 3.400357e+06 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0 disk.rbd1.out‚ 6.95703125! proc.memory.rssMB‚ 2952.328125 disk.sda3.out‚ 378.41796875 disk.sda1.out‚0 disk.sda.out‚ 378.41796875 disk.sda2.out‚0( disk.nvme0c0n1.out‚71621.69873046875$ memory_percent‚97.96819686889648$ disk./.usageGB‚286.9661979675293# disk.dm-0.out‚71621.69873046875 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125* proc.memory.percent‚18.019580841064453 network.sent‚ 2.560266e+07) disk./.usagePercent‚19.25854232627861& disk.nvme0n1.out‚71621.69873046875 disk.sda2.in‚0 disk.dm-1.out‚ 378.41796875 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 cpu‚25.283794722607166 disk.sda1.in‚0 proc.cpu.threads‚17…÷L84ã'/ _wandbruntime‚10126  _runtime‚101267çªlËä'Å ' lr-AdamW/pg1‚1.7251499794714473e-05 ' lr-AdamW/pg2‚1.7251499794714473e-05 trainer/global_step‚10049 % _timestamp‚1.7884381477953434e+09 _runtime‚10126.150433338 _step‚824„`þÐå'Ê ' lr-AdamW/pg1‚1.7251499794714473e-05 ' lr-AdamW/pg2‚1.7251499794714473e-05 trainer/global_step‚10049 % _timestamp‚1.7884381477953434e+09 _runtime‚10126.150433338 _step‚824¸ØÆkl4æ'/ _wandbruntime‚10126  _runtime‚10126¿­Maç'žP QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180365109 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 % _timestamp‚1.7884381477960374e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21484003176960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180365103 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107917290 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1197518821847040 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1197518738698240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1366711737389056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29791507871232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1167727258961408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1366711710997504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180365109 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29791562885632 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1364739520809349 trainer/global_step‚10049 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21480897783296 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216513304 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1343294960851968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMon«õÆéÃitor.on_train_batch_start/active.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1388192608780800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21484003176960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216513304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1343294986279936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1366711737389056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _runtime‚10126.151236865 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1366711710997504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 396878407 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186919807 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1364742649706241 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216536410 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180365103 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 396901519 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1388192608780800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21484003176960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216513304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1343294986279936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1366711737389056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 _runtime‚10126.151236865 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1366711710997504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 396878407 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186919807 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1364742649706241 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216536410 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180365103 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 396901519 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1388329346981376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1388329346981376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29794436279808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216557526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29794491280896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1364874155500765 trainer/global_step‚10049 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107927540 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1167839641691136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1167839669825536 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1366843231477248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21486115504128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79010069 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21483010097152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186937609 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107927543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1343424357853184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79010574 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180382393 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180382393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216534394 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21449772219613 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1366843205085696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1366843205085696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396939919 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1343424383281152 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1388326215182848 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180382387 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21446668750481 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21483010097152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396939919 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216557526 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396916781 $ _timestamp‚1.788438148690586e+09 _runtime‚10127.046595877 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21486115504128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 _step‚826 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216534394 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971_ ë'¬N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107927540 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1167839641691136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1167839669825536 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1366843231477248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21486115504128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79010069 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21483010097152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186937609 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107927543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1343424357853184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79010574 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180382393 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180382393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216534394 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21449772219613 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1366843205085696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1366843205085696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396939919 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1343424383281152 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1388326215182848 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180382387 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21446668750481 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21483010097152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396939919 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216557526 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396916781 $ _timestamp‚1.788438148690586e+09 _runtime‚10127.046595877 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21486115504128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 _step‚826 UHDeviceStatsMonitor.on_train_bÚá+y¢atch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216534394 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1388329346981376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1388329346981376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29794436279808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216557526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29794491280896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1364874155500765 trainer/global_step‚10049 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132ºjø,4ì'/ _wandbruntime‚10127  _runtime‚10127gÿaí'Û .loss/FBt1/detection_loss‚0.668432841993502 .loss/FBt2/detection_loss‚2.539540252515248 &loss/total_loss‚40.772413120461394 1loss/FBt3/assignment_loss‚0.16473969368955182 _runtime‚10127.047766075 _step‚827 1loss/SRqqt4/assignment_loss‚1.446570937314795 1loss/SRqqt4/detection_loss‚3.1900519794887967 % _timestamp‚1.7884381486927884e+09 0loss/FRt1/assignment_loss‚1.4409600823156294 0loss/FRt4/assignment_loss‚3.5076898611508884 2loss/SRqqt1/assignment_loss‚0.7934194229204546 1loss/FBt4/assignment_loss‚0.11769737559370697 1loss/FBt1/assignment_loss‚0.07035960312713846 1loss/SRqqt2/detection_loss‚2.2027915273651932 0loss/SRqqt3/detection_loss‚1.709597234725952 0loss/FRt2/assignment_loss‚3.2828500937847864 /loss/FRt2/detection_loss‚2.3947155695585978 /loss/FBt3/detection_loss‚2.6106462299823763 epoch‚8 /loss/FRt3/assignment_loss‚2.618435033057865 2loss/SRqqt2/assignment_loss‚1.2179533658605632 1loss/FBt2/assignment_loss‚0.12218613602505814 /loss/FRt1/detection_loss‚1.3782328764397291 /loss/FRt3/detection_loss‚2.5644625001832058 /loss/FBt4/detection_loss‚1.7359657565752664 trainer/global_step‚10049 2loss/SRqqt3/assignment_loss‚0.8864281286671758 /loss/FRt4/detection_loss‚3.1635104417800903 1loss/SRqqt1/detection_loss‚0.9451761763458225¢}Jˆfî'à 1loss/SRqqt4/assignment_loss‚1.446570937314795 1loss/SRqqt4/detection_loss‚3.1900519794887967 % _timestamp‚1.7884381486927884e+09 0loss/FRt1/assignment_loss‚1.4409600823156294 0loss/FRt4/assignment_loss‚3.5076898611508884 2loss/SRqqt1/assignment_loss‚0.7934194229204546 1loss/FBt4/assignment_loss‚0.11769737559370697 1loss/FBt1/assignment_loss‚0.07035960312713846 1loss/SRqqt2/detection_loss‚2.2027915273651932 0loss/SRqqt3/detection_loss‚1.709597234725952 0loss/FRt2/assignment_loss‚3.2828500937847864 /loss/FRt2/detection_loss‚2.3947155695585978 /loss/FBt3/detection_loss‚2.6106462299823763 epoch‚8 /loss/FRt3/assignment_loss‚2.618435033057865 2loss/SRqqt2/assignment_loss‚1.2179533658605632 1loss/FBt2/assignment_loss‚0.12218613602505814 /loss/FRt1/detection_loss‚1.3782328764397291 /loss/FRt3/detection_loss‚2.5644625001832058 /loss/FBt4/detection_loss‚1.7359657565752664 trainer/global_step‚10049 2loss/SRqqt3/assignment_loss‚0.8864281286671758 /loss/FRt4/detection_loss‚3.1635104417800903 1loss/SRqqt1/detection_loss‚0.9451761763458225 .loss/FBt1/detection_loss‚0.668432841993502 .loss/FBt2/detection_loss‚2.539540252515248 &loss/total_loss‚40.772413120461394 1loss/FBt3/assignment_loss‚0.16473969368955182 _runtime‚10127.047766075 _step‚827»%ƒ½ä“ï'‚(:ˆ ‘ÍåÔ³¼žµ gpu.0.gpu‚87.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚295.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u²È:{ð'‚(:ð ‘ÍåÔÙùÖ· disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 378.91015625" disk.nvme0c0n1.in‚ 471.90234375! proc.memory.rssMB‚ 2952.328125) disk./.usagePercent‚19.25968207182105 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚71685.09521484375 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.in‚0.203125 disk.rbd0.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚71685.09521484375 disk.drbd1716.in‚0& disk.nvme0n1.out‚71685.09521484375 disk.rbd1.out‚ 6.9892578125' proc.memory.availableMB‚ 332.3359375% disk./.usageGB‚286.98318099975586 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 378.91015625$ memory_percent‚97.97158241271973* proc.memory.percent‚18.019580841064453 network.sent‚ 2.5608972e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.out‚ 378.91015625 cpu‚25.283443272749334 proc.cpu.threads‚17 disk.sda3.in‚0.203125 network.recv‚ 3.405219e+06 disk.sda1.out‚0!À"¼“ñ'‚(:ˆ  ÍåÔèħµ gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.888' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.5776 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v«y|{ò'‚(:ð  ÍåÔ µÖ¶ disk.rbd1.out‚ 6.9892578125# disk.dm-0.out‚71737.06396484375 disk.sda3.out‚ 379.32421875 cpu‚25.283104484141678 disk.drbd2265.out‚0 network.sent‚ 2.5717225e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.out‚0$ disk./.usageGB‚286.9939422607422 disk.dm-0.in‚ 471.90234375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 379.32421875* proc.memory.percent‚18.019580841064453 disk.sda.in‚0.203125* disk./.usagePercent‚19.260404269075142 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚71737.06396484375 disk.rbd0.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2952.328125 proc.cpu.threads‚17 disk.dm-1.out‚ 379.32421875 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚71737.06396484375$ memory_percent‚97.97005653381348' proc.memory.availableMB‚ 332.5859375 disk.drbd2080.in‚0 network.recv‚ 3.410872e+06 disk.rbd1.in‚0ø’üלó'‚(:‘ ¯ÍåÔúí–µ gpu.0.gpu‚24.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚295.503' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.10059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Ú‚äqô'‚(:æ ¯ÍåÔ¾‘Ù· disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚19.26087685898335! proc.memory.rssMB‚ 2952.328125" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ disk.nvme0n1.out‚71784.306640625 disk.rbd1.in‚0% disk./.usageGB‚287.00098419189453 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.dm-1.out‚ 379.9765625 disk.rbd1.out‚ 7.1455078125* proc.memory.percent‚18.019580841064453 network.recv‚ 3.415179e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 471.90234375! disk.dm-0.out‚71784.306640625$ memory_percent‚97.96860218048096( proc.memory.availableMB‚ 332.82421875 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.sda3.out‚ 379.9765625 network.sent‚ 2.572319e+07 disk.sda.out‚ 379.9765625 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.out‚0 cpu‚25.28275629514443 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚71784.306640625ø¥74õ'/ _wandbruntime‚10173  _runtime‚10173äïÉö'à & lr-AdamW/pg2‚1.643974375128071e-05 trainer/global_step‚10099 % _timestamp‚1.7884381950163789e+09 _runtime‚10173.371430389 _step‚828 & lr-AdamW/pg1‚1.643974375128071e-05ËÐçÎ÷'È trainer/global_step‚10099 % _timestamp‚1.7884381950163789e+09 _runtime‚10173.371430389 _step‚828 & lr-AdamW/pg1‚1.643974375128071e-05 & lr-AdamW/pg2‚1.643974375128071e-05¼x„4ø'/ _wandbruntime‚10173  _runtime‚10173Ogz!$(ù'žP UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181229309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1371314824116357 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1349764810912768 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21589633283584 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187813012 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1349764836340736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21553116672513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187812485 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1373286415407104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181229303 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 _step‚829 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181229309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79382548 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21589633283584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217594262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1394872943297024 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1394876075082240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1373286415407104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29938431223296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108429940 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1173347802167808 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1394872943297024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚10099 % _timestamp‚1.7884381950171266e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1373286441798656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1173347774033408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚00ï~)(ú'£P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1349764836340736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21553116672513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187812485 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1373286415407104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181229303 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 _step‚829 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181229309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79382548 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21589633283584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217594262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1394872943297024 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1394876075082240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1373286415407104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29938431223296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108429940 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1173347802167808 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1394872943297024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚10099 % _timestamp‚1.7884381950171266e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1373286441798656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1173347774033408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181229309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1371314824116357 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1349764810912768 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21589633283584 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187813012 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527½wÎö4û'/ _wandbruntime‚10174  _runtime‚10174uæ< ü'¨N `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1173460184897536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1349894207913984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108440193 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚10174.270522908 _step‚830 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21591745610752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79390746 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187830438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allwòocated‚1395009681497600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21588640203776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181246593 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181246593 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217615378 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1349894233341952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1395006549699072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29941312140288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217615378 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21591745610752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884381959146752e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217592246 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79390248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21555225465821 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1373417935886848 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1373417909495296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187830939 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398861971 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1395009681497600 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1373417935886848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398838833 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29941367141376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 trainer/global_step‚10099 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21588640203776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398838833 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1203401580173312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398861971 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181246587 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971·5¡3'ý'­N SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398861971 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181246587 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1173460184897536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1349894207913984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108440193 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚10174.270522908 _step‚830 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21591745610752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79390746 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187830438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1395009681497600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21588640203776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181246593 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181246593 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217615378 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1349894233341952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1395006549699072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29941312140288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217615378 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21591745610752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884381959146752e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217592246 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79390248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21555225465821 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1373417935886848 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1373417909495296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187830939 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398861971 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1395009681497600 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1373417935886848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398838833 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29941367141376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 trainer/global_step‚10099 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21588640203776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398838833 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1203401580173312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21692157996544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1401553277813248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218629008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1356234686401536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21695263390208 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1379861119816704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788438242261677e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182093509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 _runtime‚10220.616941836 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1401553277813248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79765452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108942587 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218652114 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21658569918721 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218652114 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1379861119816704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21692157996544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30085031127552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218629008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188708039 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚10149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182093503 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108942590 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400722511 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E¡VD©Ü ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188708590 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1178968345374208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚833 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1379861146208256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400745623 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21655466449797 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182093509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1356234660973568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1401556409598464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30085086141952 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1377893256320257 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21695263390208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1377890127423365 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0úb!((Œ(¢P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1401553277813248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79765452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108942587 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218652114 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21658569918721 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218652114 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1379861119816704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21692157996544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30085031127552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218629008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188708039 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚10149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182093503 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108942590 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400722511 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188708590 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1178968345374208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚833 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1379861146208256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400745623 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21655466449797 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182093509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1356234660973568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1401556409598464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30085086141952 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1377893256320257 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21695263390208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1377890127423365 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1379861146208256 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1209053431516160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21692157996544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1401553277813248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218629008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1356234686401536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21695263390208 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1379861119816704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788438242261677e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182093509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 _runtime‚10220.616941836 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21694270310400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188725996 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218673230 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1179080728103936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1401686884215296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400784023 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1379992640296448 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚834 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21697375717376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400784023 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218650098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218673230 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1209168673314304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1379992640296448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1378024762114781 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108952843 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1401690016013824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1401690016013824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400760885 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79773674 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1379992613904896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21697375717376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3KÞw#(­N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21694270310400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188725996 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218673230 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1179080728103936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1401686884215296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400784023 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1379992640296448 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚834 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21697375717376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400784023 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218650098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218673230 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1209168673314304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1379992640296448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1378024762114781 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108952843 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1401690016013824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1401690016013824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400760885 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79773674 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pf´w¼ool.current‚ 3103469132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1379992613904896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21697375717376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7884382431519644e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÂTïI‡4(/ _wandbruntime‚10221  _runtime‚102212ÉȪ^‘(Ø /loss/FBt2/detection_loss‚2.8003640042410956 2loss/SRqqt1/assignment_loss‚0.9494808862647531 0loss/FRt2/assignment_loss‚2.9195225636164346 2loss/SRqqt3/assignment_loss‚1.5774662579071115 /loss/SRqqt2/detection_loss‚2.31746671777783 %loss/total_loss‚41.53785507716283 % _timestamp‚1.7884382431541965e+09 _step‚835 /loss/FRt1/detection_loss‚1.9787027578246086 /loss/FRt4/detection_loss‚2.7956648508707684 1loss/SRqqt1/detection_loss‚0.8386421786425146 1loss/SRqqt3/detection_loss‚1.6278493819588964 1loss/SRqqt4/detection_loss‚2.2474299781024456 /loss/FBt1/detection_loss‚0.7181403023754189 /loss/FBt3/detection_loss‚2.0194015123627405 .loss/FBt4/detection_loss‚1.916342557966709 _runtime‚10221.509145488 /loss/FRt4/assignment_loss‚3.464542349179586 2loss/SRqqt4/assignment_loss‚1.2035912052961066 3loss/FBt1/assignment_loss‚0.0022507125757090097 0loss/FBt2/assignment_loss‚0.3630497458701332 .loss/FRt2/detection_loss‚2.375823968932742 trainer/global_step‚10149 2loss/SRqqt2/assignment_loss‚1.4754934685699868 1loss/FBt4/assignment_loss‚0.19475291240378284 epoch‚8 /loss/FBt3/assignment_loss‚0.871094773663489 .loss/FRt3/detection_loss‚2.509100240551763 0loss/FRt1/assignment_loss‚1.5030492531492354 0loss/FRt3/assignment_loss‚2.8686324970589743EGýXc’(Ý 2loss/SRqqt2/assignment_loss‚1.4754934685699868 1loss/FBt4/assignment_loss‚0.19475291240378284 epoch‚8 /loss/FBt3/assignment_loss‚0.871094773663489 .loss/FRt3/detection_loss‚2.509100240551763 0loss/FRt1/assignment_loss‚1.5030492531492354 0loss/FRt3/assignment_loss‚2.8686324970589743 /loss/FBt2/detection_loss‚2.8003640042410956 2loss/SRqqt1/assignment_loss‚0.9494808862647531 0loss/FRt2/assignment_loss‚2.9195225636164346 2loss/SRqqt3/assignment_loss‚1.5774662579071115 /loss/SRqqt2/detection_loss‚2.31746671777783 %loss/total_loss‚41.53785507716283 % _timestamp‚1.7884382431541965e+09 _step‚835 /loss/FRt1/detection_loss‚1.9787027578246086 /loss/FRt4/detection_loss‚2.7956648508707684 1loss/SRqqt1/detection_loss‚0.8386421786425146 1loss/SRqqt3/detection_loss‚1.6278493819588964 1loss/SRqqt4/detection_loss‚2.2474299781024456 /loss/FBt1/detection_loss‚0.7181403023754189 /loss/FBt3/detection_loss‚2.0194015123627405 .loss/FBt4/detection_loss‚1.916342557966709 _runtime‚10221.509145488 /loss/FRt4/assignment_loss‚3.464542349179586 2loss/SRqqt4/assignment_loss‚1.2035912052961066 3loss/FBt1/assignment_loss‚0.0022507125757090097 0loss/FBt2/assignment_loss‚0.3630497458701332 .loss/FRt2/detection_loss‚2.375823968932742 trainer/global_step‚10149Ã|ÁŸ›““(‚(:ˆ ëÍåÔµÂçµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úk}~{”(‚(:ð ëÍåÔäѶ·( disk.nvme0c0n1.out‚71997.20068359375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 cpu‚25.281371560562494 proc.cpu.threads‚17$ memory_percent‚97.97244071960449* disk./.usagePercent‚19.264525222353807% disk./.usageGB‚287.05534744262695 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.sda3.out‚ 382.10546875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.sent‚ 2.5848921e+07 disk.sda2.out‚0 disk.dm-1.out‚ 382.10546875 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375 network.recv‚ 3.433591e+06 disk.drbd2266.in‚0 disk.rbd1.out‚ 7.333984375 disk.sda.out‚ 382.10546875 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚71997.20068359375! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453 disk.sda2.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 471.90234375' proc.memory.availableMB‚ 332.1953125 disk.drbd2265.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0.203125& disk.nvme0n1.out‚71997.20068359375 disk.sda1.in‚0Ç[¶ ž•(‚(:“ úÍåÔìÅÒ¶ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚290.708' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.141600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©’‹˜x–(‚(:í úÍåÔÓÎÜ· disk.drbd2080.out‚0 disk.sda.out‚ 382.69140625 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 network.recv‚ 3.439709e+06 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 cpu‚25.28103216460494 disk.rbd0.out‚0) disk./.usagePercent‚19.26478327794832$ disk./.usageGB‚287.0591926574707" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.in‚0 disk.drbd2080.in‚0# memory_percent‚97.9694128036499( proc.memory.availableMB‚ 332.69140625 network.sent‚ 2.5959988e+07 disk.dm-1.out‚ 382.69140625( disk.nvme0c0n1.out‚72049.23583984375 disk.sda3.in‚0.203125 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2952.328125 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.out‚ 382.69140625 disk.sda.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.out‚ 7.333984375# disk.dm-0.out‚72049.23583984375& disk.nvme0n1.out‚72049.23583984375* proc.memory.percent‚18.019580841064453 disk.rbd0.in‚0‡Â£„ž—(‚(:“ ‰ÎåÔ塱µ gpu.0.gpu‚76.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.217600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Ðéq˜(‚(:æ ‰ÎåÔž™‚· network.sent‚ 2.5964164e+07* disk./.usagePercent‚19.265351102659267 disk.rbd1.out‚ 7.3662109375" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2080.in‚0$ memory_percent‚97.97098636627197! proc.memory.rssMB‚ 2952.328125 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 383.1640625 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17% disk./.usageGB‚287.06765365600586 disk.nvme0n1.in‚ 471.90234375# disk.nvme0n1.out‚72099.01171875 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0% disk.nvme0c0n1.out‚72099.01171875 cpu‚25.280674502846917 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0* proc.memory.percent‚18.019580841064453 disk.dm-0.out‚72099.01171875 disk.sda3.out‚ 383.1640625 disk.drbd2080.out‚0( proc.memory.availableMB‚ 332.43359375 network.recv‚ 3.444623e+06 disk.dm-1.out‚ 383.1640625 disk.drbd2266.out‚0 disk.sda.in‚0.203125íi¦4™(/ _wandbruntime‚10267  _runtime‚10267ç¹ÈËš(Å _step‚836 ' lr-AdamW/pg1‚1.4870067306450767e-05 ' lr-AdamW/pg2‚1.4870067306450767e-05 trainer/global_step‚10199 % _timestamp‚1.7884382892140975e+09 _runtime‚10267.569116643g®Ð›(Ê ' lr-AdamW/pg1‚1.4870067306450767e-05 ' lr-AdamW/pg2‚1.4870067306450767e-05 trainer/global_step‚10199 % _timestamp‚1.7884382892140975e+09 _runtime‚10267.569116643 _step‚836Ä°XÙ4œ(/ _wandbruntime‚10267  _runtime‚10267e·{'$((žP =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219686860 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1386435824226304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1362704511034368 trainer/global_step‚10199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21797788103168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21760919696005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1386435850617856 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219709966 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109455240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1384465430730373 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109455237 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1362704536462336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1408236744114688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182957709 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21764023164929 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219686860 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1408236744114688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21800893496832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1214820314686464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219709966 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182957703 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189603419 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189602881 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30231426105856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182957709 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1184588888580608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182957703 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402667675 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 _runtime‚10267.570030428 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21800893496832 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1214820233634816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1408233612329472…> U)(ž(£P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1408236744114688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21800893496832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1214820314686464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219709966 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182957703 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189603419 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189602881 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30231426105856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182957709 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1184588888580608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182957703 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402667675 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 _runtime‚10267.570030428 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21800893496832 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1214820233634816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1408233612329472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219686860 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1386435824226304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1362704511034368 trainer/global_step‚10199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21797788103168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21760919696005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52917248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1386435850617856 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219709966 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109455240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1384465430730373 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109455237 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1362704536462336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1408236744114688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182957709 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21764023164929 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219686860 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ÅûØ€4Ÿ(/ _wandbruntime‚10268  _runtime‚10268!5W¥  (¨N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1408367218731520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80155304 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1184701271310336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1386567318314496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402682937 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1362833908035584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1386567344706048 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1184701299444736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219731082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182974993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402706075 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21803005824000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1386567344706048 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1384596936524689 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884382901058192e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30234338274304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1408370350530048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚10199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189620794 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402706075 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21799900417024 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¨)3'¡(­N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189620794 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402706075 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21799900417024 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1408367218731520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80155304 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1184701271310336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1386567318314496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402682937 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1362833908035584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1386567344706048 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1184701299444736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219731082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182974993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402706075 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21803005824000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1386567344706048 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1384596936524689 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 % _timestamp‚1.7884382901058192e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30234338274304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1408370350530048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 trainer/global_step‚10199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Ævh` 4¢(/ _wandbruntime‚10268  _runtime‚10268Ybå2_£(Ù 0loss/SRqqt3/detection_loss‚1.781137366994069 _runtime‚10268.462477728 /loss/FRt2/assignment_loss‚3.198175705614544 2loss/SRqqt1/assignment_loss‚0.8000788887341818 1loss/FBt4/assignment_loss‚0.28682752906853337 0loss/FRt1/assignment_loss‚1.3616936717714583 0loss/FRt3/assignment_loss‚2.8933011263608934 .loss/FRt3/detection_loss‚2.429050699621439 /loss/FBt4/detection_loss‚1.9632679621378581 &loss/total_loss‚40.751906011868485 epoch‚8 1loss/SRqqt4/assignment_loss‚1.443155992632875 0loss/SRqqt1/detection_loss‚0.933695277604072 0loss/SRqqt4/detection_loss‚2.408461895126563 /loss/FBt3/detection_loss‚3.6548363715410233 % _timestamp‚1.7884382901075315e+09 1loss/SRqqt2/detection_loss‚2.0738462465150014 0loss/FBt1/detection_loss‚0.45281669230435984 trainer/global_step‚10199 0loss/FBt2/assignment_loss‚0.3284639216028154 .loss/FRt4/detection_loss‚3.219576282160623 /loss/FBt2/detection_loss‚2.1699248611927033 1loss/FBt3/assignment_loss‚0.07596614738577046 _step‚839 0loss/FRt4/assignment_loss‚3.3266381344624927 2loss/SRqqt2/assignment_loss‚1.1279387036072357 2loss/SRqqt3/assignment_loss‚1.1557085680184314 2loss/FBt1/assignment_loss‚0.001151835437141727 /loss/FRt1/detection_loss‚1.4080684403814967 .loss/FRt2/detection_loss‚2.258123691592899ÿMk8d¤(Þ 1loss/SRqqt2/detection_loss‚2.0738462465150014 0loss/FBt1/detection_loss‚0.45281669230435984 trainer/global_step‚10199 0loss/FBt2/assignment_loss‚0.3284639216028154 .loss/FRt4/detection_loss‚3.219576282160623 /loss/FBt2/detection_loss‚2.1699248611927033 1loss/FBt3/assignment_loss‚0.07596614738577046 _step‚839 0loss/FRt4/assignment_loss‚3.3266381344624927 2loss/SRqqt2/assignment_loss‚1.1279387036072357 2loss/SRqqt3/assignment_loss‚1.1557085680184314 2loss/FBt1/assignment_loss‚0.001151835437141727 /loss/FRt1/detection_loss‚1.4080684403814967 .loss/FRt2/detection_loss‚2.258123691592899 0loss/SRqqt3/detection_loss‚1.781137366994069 _runtime‚10268.462477728 /loss/FRt2/assignment_loss‚3.198175705614544 2loss/SRqqt1/assignment_loss‚0.8000788887341818 1loss/FBt4/assignment_loss‚0.28682752906853337 0loss/FRt1/assignment_loss‚1.3616936717714583 0loss/FRt3/assignment_loss‚2.8933011263608934 .loss/FRt3/detection_loss‚2.429050699621439 /loss/FBt4/detection_loss‚1.9632679621378581 &loss/total_loss‚40.751906011868485 epoch‚8 1loss/SRqqt4/assignment_loss‚1.443155992632875 0loss/SRqqt1/detection_loss‚0.933695277604072 0loss/SRqqt4/detection_loss‚2.408461895126563 /loss/FBt3/detection_loss‚3.6548363715410233 % _timestamp‚1.7884382901075315e+09ÇÛué$¥(‚(:’ ˜ÎåÔ«Ò…µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.27680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݤåèr¦(‚(:ç ˜ÎåÔÔ˜™· disk.rbd1.out‚ 7.3662109375 disk.drbd1716.out‚0 disk.rbd0.out‚0$ memory_percent‚97.97248840332031 proc.cpu.threads‚17$ disk./.usageGB‚287.0817756652832 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 383.671875 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 471.90234375* proc.memory.percent‚18.019580841064453 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚72141.11865234375 disk.sda3.in‚0.203125! proc.memory.rssMB‚ 2952.328125 cpu‚25.280361999855764 network.recv‚ 3.448248e+06* disk./.usagePercent‚19.266298842551024 disk.dm-1.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚332.1875 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚72141.11865234375 disk.sda1.in‚0 disk.sda.in‚0.203125 disk.dm-1.out‚ 383.671875 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚72141.11865234375 network.sent‚ 2.5969503e+07 disk.sda.out‚ 383.671875 disk.drbd2265.in‚0W™™ž§(‚(:“ §ÎåÔðÖ¶ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚260.256' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.051199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}t0L}¨(‚(:ò §ÎåÔþ•™·* disk./.usagePercent‚19.266830314192106 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 332.41796875 network.sent‚ 2.6078582e+07 disk.rbd1.out‚ 7.6943359375 disk.drbd2265.out‚0* proc.memory.percent‚18.019580841064453 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚72202.54541015625# disk.dm-0.out‚72202.54541015625 disk.rbd0.out‚0 network.recv‚ 3.454821e+06 disk.drbd2266.out‚0$ memory_percent‚97.97108173370361! proc.memory.rssMB‚ 2952.328125 disk.drbd2265.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 384.51171875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.nvme0n1.in‚ 471.90234375% disk./.usageGB‚287.08969497680664 disk.drbd2080.out‚0 disk.sda.in‚0.203125& disk.nvme0n1.out‚72202.54541015625 disk.sda2.in‚0 disk.sda3.out‚ 384.51171875 disk.sda.out‚ 384.51171875 cpu‚25.279978492396836Dùñ’©(‚(:‡ ¶ÎåÔå¡‹µ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.791 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e¯è8uª(‚(:ê ¶ÎåÔû£Å¶ disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda.out‚ 384.953125% disk./.usageGB‚287.09594345092773 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 471.90234375& disk.nvme0n1.out‚72245.23291015625 disk.sda1.in‚0 cpu‚25.27964793444773 disk.rbd0.in‚0 disk.nvme0n1.in‚ 471.90234375" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125! proc.memory.rssMB‚ 2952.328125 network.sent‚ 2.6084418e+07 network.recv‚ 3.458795e+06$ memory_percent‚97.97112941741943( proc.memory.availableMB‚ 332.41015625 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 384.953125 disk.rbd1.out‚ 7.6943359375( disk.nvme0c0n1.out‚72245.23291015625 disk.sda3.in‚0.203125) disk./.usagePercent‚19.26724965453319 disk.sda2.out‚0# disk.dm-0.out‚72245.23291015625 disk.sda3.out‚ 384.953125* proc.memory.percent‚18.019580841064453 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0ºqˆ®4«(/ _wandbruntime‚10314  _runtime‚10314è>Üˬ(Å _step‚840 ' lr-AdamW/pg1‚1.4112474023272972e-05 ' lr-AdamW/pg2‚1.4112474023272972e-05 trainer/global_step‚10249 % _timestamp‚1.7884383362628708e+09 _runtime‚10314.617778457¤ üЭ(Ê ' lr-AdamW/pg1‚1.4112474023272972e-05 ' lr-AdamW/pg2‚1.4112474023272972e-05 trainer/global_step‚10249 % _timestamp‚1.7884383362628708e+09 _runtime‚10314.617778457 _step‚840Èç@¯ú4®(/ _wandbruntime‚10314  _runtime‚10314MâÞg¯(žP _step‚841 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1391040734037381 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1220587229233664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1369174361095168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220744712 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1414913946845696 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1414917078630912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1190209403652608 trainer/global_step‚10249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183821903 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21906523603456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404589727 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1391043862934273 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1369174386523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21903418209792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109967890 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21906523603456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183821909 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190495999 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220744712 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884383362634952e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80528639 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21869476411137 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1414913946845696 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220767818 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21903418209792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30377880595456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183821909 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220744712 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884383362634952e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80528639 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21869476411137 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1414913946845696 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220767818 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21903418209792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30377880595456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183821909 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21906523603456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404589727 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1391043862934273 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1369174386523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21903418209792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6Éó¡ïû4±(/ _wandbruntime‚10315  _runtime‚10315Ë!#-'²(§N dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1190321842651136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚10249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80536373 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109978140 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30380820873216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190513981 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21868481735313 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220765802 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80535841 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183839187 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1393142022724096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183839193 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚10315.535532163 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21908635930624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220788934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1391172239831697 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1415047553247744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1220702578291712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1415050685046272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21908635930624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109978143 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1369303783524352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 404628127 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21905530523648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1190321814516736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404604989 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1415050685046272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404628127 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183839193 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183839187 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220765802 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1220702663524352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1393142022724096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700teÏLM&³(¬N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183839193 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183839187 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220765802 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1220702663524352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1393142022724096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1190321842651136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 trainer/global_step‚10249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80536373 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109978140 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30380820873216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190513981 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21868481735313 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220765802 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80535841 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183839187 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1393142022724096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183839193 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚10315.535532163 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21908635930624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220788934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1391172239831697 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1415047553247744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1220702578291712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1415050685046272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21908635930624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109978143 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1369303783524352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 404628127 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21905530523648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1190321814516736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404604989 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.Ô)Ìåon_train_batch_end/active_bytes.all.current‚ 3131798528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1415050685046272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404628127ʮȯ4´(/ _wandbruntime‚10315  _runtime‚10315y~Þcµ(Ý 0loss/FRt1/assignment_loss‚1.0723902503056313 /loss/FRt3/assignment_loss‚2.353422416223062 0loss/FRt4/assignment_loss‚3.9105210080742836 2loss/SRqqt1/assignment_loss‚0.8463886696320283 2loss/SRqqt4/assignment_loss‚1.6801417224109172 0loss/FRt2/assignment_loss‚3.5846726749882554 1loss/SRqqt1/detection_loss‚0.8757454801862147 _step‚843 /loss/FRt2/detection_loss‚2.5642018860036675 .loss/FRt4/detection_loss‚3.054271787405014 %loss/total_loss‚40.80030632921482 1loss/SRqqt2/detection_loss‚2.3222457846005757 .loss/FBt2/detection_loss‚2.369930070638657 trainer/global_step‚10249 1loss/SRqqt3/assignment_loss‚1.348647561460986 3loss/FBt1/assignment_loss‚0.0012521658210140512 2loss/SRqqt2/assignment_loss‚1.4365311540663241 2loss/FBt3/assignment_loss‚0.029272243047931364 /loss/FBt1/detection_loss‚0.5535033766091579 /loss/FBt3/detection_loss‚2.8311232328414917 epoch‚8 1loss/FBt4/assignment_loss‚0.13570915258605964 0loss/SRqqt4/detection_loss‚2.380498181922095 % _timestamp‚1.7884383371818166e+09 /loss/FRt3/detection_loss‚2.5047420815036103 .loss/FBt4/detection_loss‚1.780532705783844 _runtime‚10315.536801629 2loss/FBt2/assignment_loss‚0.012591194512788206 1loss/SRqqt3/detection_loss‚1.6806533984461074 /loss/FRt1/detection_loss‚1.4713181301450997S6Œwh¶(â 1loss/SRqqt2/detection_loss‚2.3222457846005757 .loss/FBt2/detection_loss‚2.369930070638657 trainer/global_step‚10249 1loss/SRqqt3/assignment_loss‚1.348647561460986 3loss/FBt1/assignment_loss‚0.0012521658210140512 2loss/SRqqt2/assignment_loss‚1.4365311540663241 2loss/FBt3/assignment_loss‚0.029272243047931364 /loss/FBt1/detection_loss‚0.5535033766091579 /loss/FBt3/detection_loss‚2.8311232328414917 epoch‚8 1loss/FBt4/assignment_loss‚0.13570915258605964 0loss/SRqqt4/detection_loss‚2.380498181922095 % _timestamp‚1.7884383371818166e+09 /loss/FRt3/detection_loss‚2.5047420815036103 .loss/FBt4/detection_loss‚1.780532705783844 _runtime‚10315.536801629 2loss/FBt2/assignment_loss‚0.012591194512788206 1loss/SRqqt3/detection_loss‚1.6806533984461074 /loss/FRt1/detection_loss‚1.4713181301450997 0loss/FRt1/assignment_loss‚1.0723902503056313 /loss/FRt3/assignment_loss‚2.353422416223062 0loss/FRt4/assignment_loss‚3.9105210080742836 2loss/SRqqt1/assignment_loss‚0.8463886696320283 2loss/SRqqt4/assignment_loss‚1.6801417224109172 0loss/FRt2/assignment_loss‚3.5846726749882554 1loss/SRqqt1/detection_loss‚0.8757454801862147 _step‚843 /loss/FRt2/detection_loss‚2.5642018860036675 .loss/FRt4/detection_loss‚3.054271787405014 %loss/total_loss‚40.80030632921482Ë‘Úï“·(‚(:ˆ ÅÎåÔטµ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãA¢ùu¸(‚(:ê ÅÎåÔÿ›Ä· disk.drbd2266.out‚0$ memory_percent‚97.97301292419434% disk./.usageGB‚287.11835861206055" disk.dm-0.out‚72282.5458984375 disk.sda3.out‚ 385.6328125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 7.8505859375" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚72282.5458984375 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 cpu‚25.279309982871435 network.recv‚ 3.463791e+06 disk.sda3.in‚0.203125% disk.nvme0n1.out‚72282.5458984375 network.sent‚ 2.6090783e+07 disk.sda.in‚0.203125 disk.sda.out‚ 385.6328125 disk.rbd0.in‚0 disk.sda2.out‚0) disk./.usagePercent‚19.26875395480439 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125' proc.memory.availableMB‚ 332.1015625! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453 disk.dm-0.in‚ 471.90234375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 385.6328125 disk.drbd2266.in‚0'Y¹(‚(:’ ÔÎåÔ줳µ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚270.533' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.10660000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó[%euº(‚(:ê ÔÎåÔФⶠnetwork.sent‚ 2.6198951e+07 disk.sda3.out‚ 386.12890625 disk.drbd2265.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 332.34765625! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.out‚ 7.8505859375" disk.nvme0c0n1.in‚ 471.90234375" disk.dm-0.out‚72326.5966796875 disk.sda3.in‚0.203125 disk.sda1.in‚0) disk./.usagePercent‚19.26971807917834 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 386.12890625 disk.drbd2265.in‚0 disk.dm-1.out‚ 386.12890625 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚72326.5966796875 disk.dm-0.in‚ 471.90234375 disk.sda.in‚0.203125% disk.nvme0n1.out‚72326.5966796875 network.recv‚ 3.469152e+06$ disk./.usageGB‚287.1327247619629 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 471.90234375" memory_percent‚97.971510887146 cpu‚25.27898187892891 disk.sda1.out‚0 proc.cpu.threads‚17ðusfž»(‚(:“ ãÎåÔ‘¢óµ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚295.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.148599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â}fºp¼(‚(:å ãÎåÔÚúɶ disk.dm-0.in‚ 471.90234375$ disk./.usageGB‚287.1466255187988 disk.drbd2265.in‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚72381.82861328125 disk.sda1.in‚0% proc.memory.availableMB‚ 332.34375 disk.nvme0n1.in‚ 471.90234375& disk.nvme0n1.out‚72381.82861328125 disk.sda3.out‚ 386.703125 network.recv‚ 3.473898e+06 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 8.0068359375 disk.sda3.in‚0.203125 disk.sda2.out‚0 network.sent‚ 2.6205116e+07) disk./.usagePercent‚19.27065097063311 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 386.703125 disk.drbd2080.in‚0 disk.sda.out‚ 386.703125 disk.drbd2265.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚72381.82861328125 cpu‚25.27865178708901 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0# memory_percent‚97.9715347290039 proc.cpu.threads‚17ň¸½4½(/ _wandbruntime‚10361  _runtime‚10361 <ãi˾(Å ' lr-AdamW/pg1‚1.3373255806804296e-05 ' lr-AdamW/pg2‚1.3373255806804296e-05 trainer/global_step‚10299 % _timestamp‚1.7884383835416968e+09 _runtime‚10361.896717436 _step‚844?!á‰Ð¿(Ê trainer/global_step‚10299 % _timestamp‚1.7884383835416968e+09 _runtime‚10361.896717436 _step‚844 ' lr-AdamW/pg1‚1.3373255806804296e-05 ' lr-AdamW/pg2‚1.3373255806804296e-05̤?g4À(/ _wandbruntime‚10361  _runtime‚10361ð9è$(Á(žP YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110480540 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1195829974993408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1399585233045504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22009048316416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 406511779 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221802564 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21974929657345 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22012153710080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22009048316416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80911725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22012153710080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191391732 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30524799271936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1421597413147136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80911195 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1397619166241281 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221802564 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1421597413147136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221825670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚10299 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1421594281361920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1375644236583936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1399585233045504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚10361.897985029 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21971826188421 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1226354774265344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1421594281361920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184686109 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1421594281361920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184686109 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80911725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22012153710080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191391732 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30524799271936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1421597413147136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80911195 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1397619166241281 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221802564 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1421597413147136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221825670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚10299 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1421594281361920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1375644236583936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1399585233045504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚10361.897985029Í þ44Ã(/ _wandbruntime‚10362  _runtime‚10362†ôú‘òÄ(¨N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 406550179 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1399716753525248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80919409 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 184703387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22014266037248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1226470120750080 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1195942357723136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221823654 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191410202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 184703393 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110490790 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1397750672035805 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 406550179 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1421731019562496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22014266037248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191409687 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 406527041 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884383844593892e+09…m«ž3'Å(­N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 406550179 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1421731019562496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22014266037248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191409687 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 406527041 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884383844593892e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 406550179 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1399716753525248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80919409 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 184703387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22014266037248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1226470120750080 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1195942357723136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221823654 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191410202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 184703393 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110490790 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1397750672035805ÎTÏT4Æ(/ _wandbruntime‚10362  _runtime‚10362)×^Ç(Ø 0loss/FBt4/assignment_loss‚0.8821365139091439 0loss/SRqqt4/detection_loss‚2.478677840743746 epoch‚8 % _timestamp‚1.7884383844620578e+09 /loss/FRt2/assignment_loss‚3.306730862174715 /loss/FRt4/assignment_loss‚2.939665007180181 2loss/SRqqt1/assignment_loss‚1.0397365104564162 1loss/FBt2/assignment_loss‚0.31389240692887044 .loss/FRt1/detection_loss‚1.570695412904024 4loss/FBt1/assignment_loss‚0.00012720624168092962 .loss/FRt2/detection_loss‚2.945017955132893 .loss/FBt3/detection_loss‚1.971782962481181 trainer/global_step‚10299 0loss/FRt3/assignment_loss‚2.2243047045035795 2loss/SRqqt4/assignment_loss‚1.1576562437627997 3loss/FBt3/assignment_loss‚0.0028289317851854573 0loss/SRqqt1/detection_loss‚0.944634084376585 .loss/FBt1/detection_loss‚0.669773838817118 %loss/total_loss‚40.13005403204019 _step‚847 0loss/FRt1/assignment_loss‚1.3469785809516908 1loss/SRqqt2/assignment_loss‚1.539309624967904 /loss/FRt3/detection_loss‚2.3789850290526044 0loss/SRqqt2/detection_loss‚2.376313551746566 .loss/FBt2/detection_loss‚2.221541871627172 /loss/FRt4/detection_loss‚2.7593982363569327 1loss/SRqqt3/detection_loss‚1.4992842683151586 2loss/SRqqt3/assignment_loss‚1.2060691133252721 /loss/FBt4/detection_loss‚2.3545132742987738 _runtime‚10362.817048307G½CcÈ(Ý .loss/FBt3/detection_loss‚1.971782962481181 trainer/global_step‚10299 0loss/FRt3/assignment_loss‚2.2243047045035795 2loss/SRqqt4/assignment_loss‚1.1576562437627997 3loss/FBt3/assignment_loss‚0.0028289317851854573 0loss/SRqqt1/detection_loss‚0.944634084376585 .loss/FBt1/detection_loss‚0.669773838817118 %loss/total_loss‚40.13005403204019 _step‚847 0loss/FRt1/assignment_loss‚1.3469785809516908 1loss/SRqqt2/assignment_loss‚1.539309624967904 /loss/FRt3/detection_loss‚2.3789850290526044 0loss/SRqqt2/detection_loss‚2.376313551746566 .loss/FBt2/detection_loss‚2.221541871627172 /loss/FRt4/detection_loss‚2.7593982363569327 1loss/SRqqt3/detection_loss‚1.4992842683151586 2loss/SRqqt3/assignment_loss‚1.2060691133252721 /loss/FBt4/detection_loss‚2.3545132742987738 _runtime‚10362.817048307 0loss/FBt4/assignment_loss‚0.8821365139091439 0loss/SRqqt4/detection_loss‚2.478677840743746 epoch‚8 % _timestamp‚1.7884383844620578e+09 /loss/FRt2/assignment_loss‚3.306730862174715 /loss/FRt4/assignment_loss‚2.939665007180181 2loss/SRqqt1/assignment_loss‚1.0397365104564162 1loss/FBt2/assignment_loss‚0.31389240692887044 .loss/FRt1/detection_loss‚1.570695412904024 4loss/FBt1/assignment_loss‚0.00012720624168092962 .loss/FRt2/detection_loss‚2.945017955132893Ï„MžÉ(‚(:“ òÎåÔñ¨µ gpu.0.gpu‚46.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.615' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.723000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àEbzÊ(‚(:ï òÎåÔúïô¶& disk.nvme0n1.out‚72425.50439453125% disk./.usageGB‚287.16698837280273 disk.sda.out‚ 387.3359375 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 8.0068359375" disk.nvme0c0n1.in‚ 471.90234375! proc.memory.rssMB‚ 2952.328125 network.sent‚ 2.6210955e+07* disk./.usagePercent‚19.272017538850882 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.dm-1.out‚ 387.3359375 disk.rbd1.in‚0 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚72425.50439453125 disk.sda3.in‚0.203125 disk.sda.in‚0.203125$ memory_percent‚97.97170162200928( proc.memory.availableMB‚ 332.31640625 cpu‚25.278319955260937 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 387.3359375 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚72425.50439453125* proc.memory.percent‚18.019580841064453 proc.cpu.threads‚17 network.recv‚ 3.477925e+06 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0°ÚÄ‘Ë(‚(:† ÏåÔ掠 gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚137.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È“^›vÌ(‚(:ë ÏåÔ…üÕ¶ disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 471.90234375* proc.memory.percent‚18.019580841064453 disk.rbd0.in‚0 disk.dm-1.out‚ 387.83984375 disk.dm-0.in‚ 471.90234375! proc.memory.rssMB‚ 2952.328125 cpu‚25.27799209584379 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.in‚0# disk.dm-0.out‚72478.32666015625 proc.cpu.threads‚17 network.recv‚ 3.48469e+06 disk.drbd2080.in‚0 disk.sda.out‚ 387.83984375' proc.memory.availableMB‚ 332.0703125$ disk./.usageGB‚287.1674919128418 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.rbd1.out‚ 8.0390625$ memory_percent‚97.97320365905762 network.sent‚ 2.6320118e+07 disk.sda3.in‚0.203125 disk.drbd2080.out‚0& disk.nvme0n1.out‚72478.32666015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚72478.32666015625* disk./.usagePercent‚19.272051331845404 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 387.83984375 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.sda.in‚0.203125€Eº4“Í(‚(:ˆ ÏåÔ«Œí´ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚175.407' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂV˜ xÎ(‚(:í ÏåÔ‘ŸÂ· disk.rbd0.in‚0 network.sent‚ 2.6325454e+07* disk./.usagePercent‚19.273151396220033 disk.sda1.out‚0 disk.drbd2080.out‚0$ memory_percent‚97.97170162200928* proc.memory.percent‚18.019580841064453$ disk./.usageGB‚287.1838836669922 disk.sda3.in‚0.203125 disk.sda.out‚ 388.26171875 disk.drbd2265.in‚0& disk.nvme0n1.out‚72518.99072265625 disk.sda2.out‚0 disk.dm-1.in‚0.203125 proc.cpu.threads‚17# disk.dm-0.out‚72518.99072265625 network.recv‚ 3.488476e+06 disk.drbd2265.out‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚72518.99072265625 disk.sda3.out‚ 388.26171875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2952.328125 cpu‚25.27765964393629 disk.rbd1.out‚ 8.0390625 disk.drbd2266.in‚0 disk.dm-1.out‚ 388.26171875( proc.memory.availableMB‚ 332.31640625*ž ç4Ï(/ _wandbruntime‚10409  _runtime‚10409 ÷møÉÐ(à trainer/global_step‚10349 % _timestamp‚1.7884384309664097e+09 _runtime‚10409.321515417 _step‚848 & lr-AdamW/pg1‚1.265256670901311e-05 & lr-AdamW/pg2‚1.265256670901311e-058'$ÎÑ(È _step‚848 & lr-AdamW/pg1‚1.265256670901311e-05 & lr-AdamW/pg2‚1.265256670901311e-05 trainer/global_step‚10349 % _timestamp‚1.7884384309664097e+09 _runtime‚10409.321515417ÐQ-A4Ò(/ _wandbruntime‚10409  _runtime‚10409£¥Jl!Ó(žP VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1404191340651397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22077279434629 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1428277747663360 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192285607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1428274615878144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81292982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30671120564224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1382114061216768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110993187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81292420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30671177675776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 408410719 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185550303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1406159937455104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222883522 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185550309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1382114086644736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185550309 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22114678423040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22117783816704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚10349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222860416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1232121610629632 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1404194469548289 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1406159937455104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22114678423040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1201450490065408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222883522 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185550303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚10409.322393484 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110993190 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚565 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406159963846656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 f¤[7!QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1201450518199808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22117783816704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1428277747663360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1428274615878144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222860416úN})(Ô(£P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81292420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30671177675776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 408410719 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185550303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1406159937455104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222883522 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185550309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1382114086644736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185550309 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22114678423040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22117783816704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 trainer/global_step‚10349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222860416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1232121610629632 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1404194469548289 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1406159937455104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22114678423040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1201450490065408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222883522 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185550303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 _runtime‚10409.322393484 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110993190 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚565 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406159963846656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1201450518199808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22117783816704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1428277747663360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1428274615878144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222860416 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1404191340651397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22077279434629 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1428277747663360 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192285607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1428274615878144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81292982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30671120564224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1382114061216768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110993187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ÑØCxŒžÕ(‚(:“ ŸÏåÔ—–Ÿµ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚282.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.531200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄGf×tÖ(‚(:é ŸÏåÔĸķ disk.sda.in‚0.203125 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 8.1953125( disk.nvme0c0n1.out‚72568.20361328125 disk.dm-0.in‚ 471.90234375* proc.memory.percent‚18.019580841064453 network.sent‚ 2.6331411e+07( proc.memory.availableMB‚ 332.80859375 cpu‚25.27733235070852) disk./.usagePercent‚19.27259457983306 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0# memory_percent‚97.9686975479126 proc.cpu.threads‚17% disk./.usageGB‚287.17558670043945 disk.drbd2080.out‚0 disk.sda.out‚ 388.8828125& disk.nvme0n1.out‚72568.20361328125 disk.drbd2266.out‚0# disk.dm-0.out‚72568.20361328125 disk.sda3.out‚ 388.8828125 disk.rbd0.out‚0 disk.dm-1.out‚ 388.8828125 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚ 3.493094e+06 disk.nvme0n1.in‚ 471.90234375" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2952.328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0Á{}Ø4×(/ _wandbruntime‚10410  _runtime‚10410[ÿ².'Ø(¨N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111003440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚30674051159040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1404322846445713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185567587 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22119896143872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1201562900929536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81300337 trainer/global_step‚10349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1406291431543296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 408472231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192303256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222881506 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1428411354078720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222904638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81299816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222881506 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1428408222280192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 408449093 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1232237037321216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1201562929063936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚850 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30674108257280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 408472231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22116790736896 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192303780 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111003443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222904638 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1428411354078720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 408449093 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚10410.204426932 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22119896143872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1201562900929536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81300337 trainer/global_step‚10349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1406291431543296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 408472231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192303256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222881506 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1428411354078720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222904638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81299816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222881506 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1428408222280192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 408449093 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1232237037321216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1201562929063936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚850 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30674108257280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 408472231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_bat[%?Ú"ch_end/active_bytes.small_pool.freed‚22116790736896 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192303780 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111003443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222904638 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1428411354078720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 408449093 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 _runtime‚10410.204426932ÒAÉk4Ú(/ _wandbruntime‚10410  _runtime‚10410Ñ¥ ]Û(× /loss/FRt1/assignment_loss‚1.214090302796431 /loss/FRt2/assignment_loss‚3.629166214089645 1loss/SRqqt4/assignment_loss‚1.461976373208348 /loss/FRt4/detection_loss‚3.0748977444388648 0loss/FBt4/assignment_loss‚0.1021585795097053 1loss/SRqqt3/detection_loss‚1.8467500106148098 trainer/global_step‚10349 0loss/SRqqt2/assignment_loss‚1.62693935384353 2loss/SRqqt3/assignment_loss‚1.7370076203192382 .loss/FRt2/detection_loss‚2.471283280535748 1loss/SRqqt1/detection_loss‚0.8559976247035794 0loss/SRqqt2/detection_loss‚2.358092878262202 /loss/FBt3/detection_loss‚2.2171866042273387 0loss/FRt3/assignment_loss‚2.2952109426259995 3loss/FBt1/assignment_loss‚0.0004221211989655021 epoch‚8 $ _timestamp‚1.788438431850636e+09 2loss/SRqqt1/assignment_loss‚1.0001989314447584 1loss/FBt2/assignment_loss‚0.17684031014020243 1loss/FBt3/assignment_loss‚0.15106119292977382 1loss/SRqqt4/detection_loss‚2.1578644669868727 /loss/FBt2/detection_loss‚2.6722063289748297 /loss/FBt4/detection_loss‚3.0406164990531073 %loss/total_loss‚41.41090026700202 _runtime‚10410.20564439 /loss/FRt1/detection_loss‚1.4434982640012888 /loss/FBt1/detection_loss‚0.6063545832488065 _step‚851 0loss/FRt4/assignment_loss‚2.6518410552631724 /loss/FRt3/detection_loss‚2.6192389845848085?¹ÉwbÜ(Ü 1loss/FBt3/assignment_loss‚0.15106119292977382 1loss/SRqqt4/detection_loss‚2.1578644669868727 /loss/FBt2/detection_loss‚2.6722063289748297 /loss/FBt4/detection_loss‚3.0406164990531073 %loss/total_loss‚41.41090026700202 _runtime‚10410.20564439 /loss/FRt1/detection_loss‚1.4434982640012888 /loss/FBt1/detection_loss‚0.6063545832488065 _step‚851 0loss/FRt4/assignment_loss‚2.6518410552631724 /loss/FRt3/detection_loss‚2.6192389845848085 /loss/FRt1/assignment_loss‚1.214090302796431 /loss/FRt2/assignment_loss‚3.629166214089645 1loss/SRqqt4/assignment_loss‚1.461976373208348 /loss/FRt4/detection_loss‚3.0748977444388648 0loss/FBt4/assignment_loss‚0.1021585795097053 1loss/SRqqt3/detection_loss‚1.8467500106148098 trainer/global_step‚10349 0loss/SRqqt2/assignment_loss‚1.62693935384353 2loss/SRqqt3/assignment_loss‚1.7370076203192382 .loss/FRt2/detection_loss‚2.471283280535748 1loss/SRqqt1/detection_loss‚0.8559976247035794 0loss/SRqqt2/detection_loss‚2.358092878262202 /loss/FBt3/detection_loss‚2.2171866042273387 0loss/FRt3/assignment_loss‚2.2952109426259995 3loss/FBt1/assignment_loss‚0.0004221211989655021 epoch‚8 $ _timestamp‚1.788438431850636e+09 2loss/SRqqt1/assignment_loss‚1.0001989314447584 1loss/FBt2/assignment_loss‚0.17684031014020243ÓÏFPÊ“Ý(‚(:ˆ ®ÏåÔôõµ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚266.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îꤎoÞ(‚(:ä ®ÏåÔïÒì¶ disk.rbd0.in‚0 disk.drbd2266.out‚0% disk.nvme0c0n1.out‚72611.66015625' proc.memory.availableMB‚ 332.7578125 proc.cpu.threads‚17% disk./.usageGB‚287.20061111450195 network.recv‚ 3.499225e+06 disk.sda3.in‚0.203125 disk.sda.out‚ 389.34765625 disk.dm-1.out‚ 389.34765625 disk.sda3.out‚ 389.34765625 disk.drbd2265.in‚0# disk.nvme0n1.out‚72611.66015625$ memory_percent‚97.96900749206543* proc.memory.percent‚18.019580841064453 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 471.90234375! proc.memory.rssMB‚ 2952.328125 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.dm-0.out‚72611.66015625 cpu‚25.277009472243684 network.sent‚ 2.6428792e+07 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 disk.rbd1.out‚ 8.1953125) disk./.usagePercent‚19.27427398925768 disk.sda1.out‚0 disk.drbd2080.in‚0Å_ÿ“ß(‚(:ˆ ½ÏåÔÀ©¾µ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚263.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á#àË|à(‚(:ñ ½ÏåÔ’üî¶ disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125$ memory_percent‚97.96898365020752! proc.memory.rssMB‚ 2952.328125 disk.sda2.in‚0( disk.nvme0c0n1.out‚72662.40283203125 disk.drbd2080.in‚0 disk.sda.out‚ 389.90234375 cpu‚25.27668400967485% disk./.usageGB‚287.21645736694336 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme0n1.out‚72662.40283203125* proc.memory.percent‚18.019580841064453 network.sent‚ 2.6527455e+07 network.recv‚ 3.505888e+06* disk./.usagePercent‚19.275337444554914 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 389.90234375" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚72662.40283203125 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.sda3.in‚0.203125 disk.sda3.out‚ 389.90234375 disk.rbd1.out‚ 8.2275390625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 332.76171875y ¥Pžá(‚(:“ ÌÏåÔ…ýµ gpu.0.gpu‚75.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚295.847' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.169399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µA¾tâ(‚(:é ÌÏåÔ¡ƒî¶ disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 471.90234375 cpu‚25.276359623742476 network.sent‚ 2.6532774e+07$ disk./.usageGB‚287.2300605773926# disk.dm-0.out‚72704.87548828125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 8.2275390625 disk.drbd2080.out‚0 disk.sda3.out‚ 390.359375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 332.7578125! proc.memory.rssMB‚ 2952.328125 network.recv‚ 3.50925e+06 disk.dm-0.in‚ 471.90234375 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚72704.87548828125 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚72704.87548828125 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.out‚ 390.359375 disk.drbd1716.out‚0 disk.sda.out‚ 390.359375 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0* disk./.usagePercent‚19.276250367422012 disk.drbd1716.in‚0 disk.sda.in‚0.203125$ memory_percent‚97.96900749206543* proc.memory.percent‚18.019580841064453 disk.sda3.in‚0.203125 disk.drbd2265.in‚0õMë_4ã(/ _wandbruntime‚10456  _runtime‚10456Ù¸lÊä(Ä ' lr-AdamW/pg1‚1.1950556920427136e-05 ' lr-AdamW/pg2‚1.1950556920427136e-05 trainer/global_step‚10399 $ _timestamp‚1.788438477976928e+09 _runtime‚10456.331937804 _step‚8526i–Ïå(É ' lr-AdamW/pg1‚1.1950556920427136e-05 ' lr-AdamW/pg2‚1.1950556920427136e-05 trainer/global_step‚10399 $ _timestamp‚1.788438477976928e+09 _runtime‚10456.331937804 _step‚852Ô¨|Ì 4æ(/ _wandbruntime‚10456  _runtime‚10456¿+ù$(ç(žP XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22220308529664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223918268 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚853 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22223413923328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22220308529664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111505837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30817800382464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884384779776442e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186414503 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22182732680837 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223918268 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1410766643958405 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186414503 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 410355883 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193178896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1434954950394368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 _runtime‚10456.332879963 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 410332771 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186414509 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1434958082179584 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1237888861788672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30817745368064 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1388583936705536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10399 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1412734668256256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193179411 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1207071033271808 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111505840 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1412734641864704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186414509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1434958082179584 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1434954950394368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1410769772855297 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186414509 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1434958082179584 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1237888861788672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30817745368064 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1388583936705536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10399 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1412734668256256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193179411 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1207071033271808 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111505840 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1412734641864704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186414509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1434958082179584 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1434954950394368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1410769772855297 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111505837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30817800382464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884384779776442e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186414503 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22182732680837 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223918268 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1410766643958405 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186414503 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 410355883 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193178896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1434954950394368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217ÕGœx4é(/ _wandbruntime‚10457  _runtime‚10457!èýÑÈ ê(¨N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223939358 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 410394283 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193196801 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1238004142080000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186431787 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1435088556796416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111516090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81680711 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1238004223118336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22222420843520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 410371145 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1412866135952896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223962490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22184841473937 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186431793 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1412866162344448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1410898149752721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223939358 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186431793 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 410394283 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884384788743117e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1412866162344448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22225526250496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440Öo‡3'ë(­N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1435088556796416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111516090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81680711 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1238004223118336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22222420843520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 410371145 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1412866135952896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223962490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22184841473937 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186431793 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1412866162344448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1410898149752721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223939358 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186431793 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 410394283 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884384788743117e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1412866162344448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22225526250496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223939358 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 410394283 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193196801 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1238004142080000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186431787 Qcî(Ý 0loss/FBt2/assignment_loss‚0.2564268965274096 /loss/FRt1/detection_loss‚1.6067364445977634 .loss/FRt4/detection_loss‚2.617682234807448 _runtime‚10457.231423735 _step‚855 0loss/FRt2/assignment_loss‚2.5863562747836113 /loss/FRt3/assignment_loss‚2.862633716118963 2loss/FBt1/assignment_loss‚0.006952300661131794 epoch‚8 2loss/SRqqt1/assignment_loss‚0.9020883670099169 1loss/FBt4/assignment_loss‚0.04588782117628541 /loss/FRt3/detection_loss‚2.5972289372431603 /loss/FBt1/detection_loss‚0.5523148509208113 .loss/FBt4/detection_loss‚2.155110183515047 %loss/total_loss‚38.75943069160123 2loss/SRqqt2/assignment_loss‚1.1740079030394555 1loss/SRqqt3/detection_loss‚1.7193242024492335 trainer/global_step‚10399 /loss/FRt2/detection_loss‚2.4245201006531714 0loss/FRt4/assignment_loss‚3.1713716622554893 2loss/SRqqt3/assignment_loss‚1.2537771844201617 .loss/FBt3/detection_loss‚2.450618803501129 % _timestamp‚1.7884384788764126e+09 0loss/FRt1/assignment_loss‚1.2587321410256047 1loss/SRqqt1/detection_loss‚0.8482360733022173 2loss/SRqqt4/assignment_loss‚1.2766845392945565 0loss/FBt3/assignment_loss‚0.2826110895572356 1loss/SRqqt2/detection_loss‚2.1401250345366343 0loss/SRqqt4/detection_loss‚2.080684820288106 .loss/FBt2/detection_loss‚2.489319109916687×H²)’ï(‚(:‡ ÛÏåÔüÆÚµ gpu.0.gpu‚51.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚296.191' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.2382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Ýwð(‚(:ì ÛÏåÔû¢£¸ disk.dm-1.out‚ 390.93359375 disk.rbd1.out‚ 8.3837890625 disk.dm-0.in‚ 471.90234375 disk.drbd2265.in‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2952.328125 disk.sda3.out‚ 390.93359375 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.in‚0.203125* proc.memory.percent‚18.019580841064453 disk.rbd1.in‚0 disk.sda.out‚ 390.93359375 disk.rbd0.in‚0( proc.memory.availableMB‚ 332.73046875( disk.nvme0c0n1.out‚72752.28662109375 disk.sda2.out‚0 network.recv‚ 3.513989e+06 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0& disk.nvme0n1.out‚72752.28662109375 disk.sda2.in‚0 network.sent‚ 2.6539044e+07) disk./.usagePercent‚19.27726082915966 disk.drbd1716.out‚0 disk.drbd2266.in‚0# memory_percent‚97.9691743850708 cpu‚25.276052794671703 proc.cpu.threads‚17! disk./.usageGB‚287.2451171875" disk.nvme0c0n1.in‚ 471.90234375# disk.dm-0.out‚72752.28662109375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125!!¸žñ(‚(:“ êÏåÔÑÏô¶ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚304.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.893600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"¬…qsò(‚(:è êÏåÔÏÑñ¸ disk.drbd2080.in‚0 cpu‚25.275703348805916% disk./.usageGB‚287.26087188720703 disk.rbd0.in‚0 disk.drbd2266.in‚0 network.sent‚ 2.6648065e+07 disk.sda2.out‚0# disk.dm-0.out‚72793.42333984375 disk.sda.in‚0.203125 disk.sda.out‚ 391.453125 network.recv‚ 3.520178e+06) disk./.usagePercent‚19.27831814027607 disk.drbd2265.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚72793.42333984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 391.453125 disk.rbd1.in‚0 disk.dm-0.in‚ 471.90234375 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 471.90234375& disk.nvme0n1.out‚72793.42333984375" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.in‚0% proc.memory.availableMB‚ 332.96875! proc.memory.rssMB‚ 2952.328125* proc.memory.percent‚18.019580841064453 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 391.453125$ memory_percent‚97.96772003173828 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 8.3837890625 disk.sda3.in‚0.203125×Cwn’ó(‚(:‡ ùÏåÔ±‘‡µ gpu.0.gpu‚79.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«¦Êzô(‚(:ï ùÏåÔ­Ðß·" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375* proc.memory.percent‚18.019580841064453* disk./.usagePercent‚19.278696263404143 disk.sda3.out‚ 392.17578125 disk.dm-1.out‚ 392.17578125 disk.rbd1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚287.26650619506836 disk.sda.out‚ 392.17578125 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 8.416015625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0$ memory_percent‚97.97220230102539& proc.memory.availableMB‚ 332.234375 cpu‚25.275411270821944 network.recv‚ 3.524732e+06 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚72842.11669921875# disk.dm-0.out‚72842.11669921875! proc.memory.rssMB‚ 2952.328125 network.sent‚ 2.6654302e+07 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125 disk.drbd2265.out‚0& disk.nvme0n1.out‚72842.11669921875 disk.rbd0.out‚0 disk.sda2.in‚0kŸ Ê4õ(/ _wandbruntime‚10503  _runtime‚10503qÑŸËö(Å _runtime‚10503.584032813 _step‚856 ' lr-AdamW/pg1‚1.1267372738834151e-05 ' lr-AdamW/pg2‚1.1267372738834151e-05 trainer/global_step‚10449 % _timestamp‚1.7884385252289891e+09ÔƉÐ÷(Ê ' lr-AdamW/pg2‚1.1267372738834151e-05 trainer/global_step‚10449 % _timestamp‚1.7884385252289891e+09 _runtime‚10503.584032813 _step‚856 ' lr-AdamW/pg1‚1.1267372738834151e-05Øë-iy4ø(/ _wandbruntime‚10503  _runtime‚10503_ÉÍÃÙù(P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187278703 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224976120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1417345076162305 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194073036 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1419309372665856 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1419309346274304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 412277935 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112018490 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22325938636288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1417341947265413 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 412254823 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187278703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187278709 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1243655815223808 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 trainer/global_step‚10449 $ _timestamp‚1.788438525229717e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30964295857152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1441638416695808 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1441635284910592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1212691576478208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚857 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1212691604612608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1395053786766336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224976120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_trainJCMJ _batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112018487 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82054546 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1441635284910592 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1243655900469760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1419309346274304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187278709 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1419309372665856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194072478 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952íO…"((ú(¢P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1419309346274304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187278709 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1419309372665856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194072478 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187278703 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224976120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1417345076162305 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194073036 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1419309372665856 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1419309346274304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 412277935 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112018490 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22325938636288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1417341947265413 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 412254823 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187278703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187278709 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1243655815223808 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 trainer/global_step‚10449 $ _timestamp‚1.788438525229717e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30964295857152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1441638416695808 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1441635284910592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1212691576478208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 _step‚857 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1212691604612608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1395053786766336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224976120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112018487 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82054546 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1441635284910592 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1243655900469760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194090261 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22328050950144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82062061 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22328050950144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194090804 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22331156357120 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225020342 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1212804015476736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 412316335 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1419440840362496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22331156357120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224997210 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1417473453059729 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 412316335 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1395183158339584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187295993 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1243771136692736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 _runtime‚10504.47307699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225020342 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112028743 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1441772023111168 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1419440840362496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 412293197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚10449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22293398189277 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22290294720145 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1212803987342336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187295987 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1243771221925376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30967206448640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180Û5É &ý(¬N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225020342 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112028743 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1441772023111168 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1419440840362496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 412293197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚10449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22293398189277 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22290294720145 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1212803987342336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187295987 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1243771221925376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30967206448640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 412293197 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224997210 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82061521 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚30967149350400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187295993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1395183183767552 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112028740 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194090261 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22328050950144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82062061 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22328050950144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194090804 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22331156357120 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225020342 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1212804015476736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 412316335 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1419440840362496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22331156357120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224997210 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1417473453059729 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 412316335 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1395183158339584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187295993 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1243771136692736 WHDeŠ¾ü‘(viceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 _runtime‚10504.47307699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240Ú¹ÿ_4þ(/ _wandbruntime‚10504  _runtime‚10504¡ÐP^aÿ(Û 0loss/SRqqt2/detection_loss‚2.593079040093081 3loss/FBt1/assignment_loss‚0.0005199862498663194 /loss/FRt3/detection_loss‚2.4627587658663592 0loss/SRqqt1/detection_loss‚0.958119602075645 0loss/SRqqt4/detection_loss‚2.422934091091156 trainer/global_step‚10449 % _timestamp‚1.7884385261199915e+09 2loss/SRqqt2/assignment_loss‚1.4610993787646294 _runtime‚10504.474967088 /loss/FRt4/assignment_loss‚2.768959495154294 2loss/SRqqt3/assignment_loss‚0.8406174149758254 %loss/total_loss‚39.79616220969004 /loss/FRt4/detection_loss‚2.7801267233761875 /loss/FBt1/detection_loss‚0.5781406496391922 0loss/FRt2/assignment_loss‚3.4130821892193386 /loss/FBt3/detection_loss‚2.9364645878473916 _step‚859 0loss/FRt1/assignment_loss‚1.4097437493262752 0loss/FRt3/assignment_loss‚2.6679540624221167 1loss/SRqqt1/assignment_loss‚0.773958374265131 0loss/FBt4/assignment_loss‚0.0906967324598764 /loss/FRt1/detection_loss‚1.6346893376640734 3loss/FBt3/assignment_loss‚0.0011649566100118136 1loss/SRqqt3/detection_loss‚1.6876424387670481 /loss/FBt2/detection_loss‚2.3999677672982216 /loss/FBt4/detection_loss‚2.0901832557641544 epoch‚8 .loss/FRt2/detection_loss‚2.529185015814645 2loss/SRqqt4/assignment_loss‚1.1433472666357245 0loss/FBt2/assignment_loss‚0.1517273283097893û–\f€)à .loss/FRt2/detection_loss‚2.529185015814645 2loss/SRqqt4/assignment_loss‚1.1433472666357245 0loss/FBt2/assignment_loss‚0.1517273283097893 0loss/SRqqt2/detection_loss‚2.593079040093081 3loss/FBt1/assignment_loss‚0.0005199862498663194 /loss/FRt3/detection_loss‚2.4627587658663592 0loss/SRqqt1/detection_loss‚0.958119602075645 0loss/SRqqt4/detection_loss‚2.422934091091156 trainer/global_step‚10449 % _timestamp‚1.7884385261199915e+09 2loss/SRqqt2/assignment_loss‚1.4610993787646294 _runtime‚10504.474967088 /loss/FRt4/assignment_loss‚2.768959495154294 2loss/SRqqt3/assignment_loss‚0.8406174149758254 %loss/total_loss‚39.79616220969004 /loss/FRt4/detection_loss‚2.7801267233761875 /loss/FBt1/detection_loss‚0.5781406496391922 0loss/FRt2/assignment_loss‚3.4130821892193386 /loss/FBt3/detection_loss‚2.9364645878473916 _step‚859 0loss/FRt1/assignment_loss‚1.4097437493262752 0loss/FRt3/assignment_loss‚2.6679540624221167 1loss/SRqqt1/assignment_loss‚0.773958374265131 0loss/FBt4/assignment_loss‚0.0906967324598764 /loss/FRt1/detection_loss‚1.6346893376640734 3loss/FBt3/assignment_loss‚0.0011649566100118136 1loss/SRqqt3/detection_loss‚1.6876424387670481 /loss/FBt2/detection_loss‚2.3999677672982216 /loss/FBt4/detection_loss‚2.0901832557641544 epoch‚8Ûžà›Š)‚(:’ ˆÐåÔ…ù£µ gpu.0.gpu‚50.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.324' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.66480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁX8'w‚)‚(:ì ˆÐåÔéÕŠ· disk.sda3.out‚ 392.52734375 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 332.0703125! proc.memory.rssMB‚ 2952.328125 disk.sda.out‚ 392.52734375" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.out‚0 disk.sda.in‚0.203125$ memory_percent‚97.97320365905762* proc.memory.percent‚18.019580841064453 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚72884.22607421875 proc.cpu.threads‚17 network.sent‚ 2.665959e+07 disk.dm-1.out‚ 392.52734375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 cpu‚25.27504629108032 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚72884.22607421875 disk.sda1.out‚0 network.recv‚ 3.528348e+06* disk./.usagePercent‚19.279296345064402# disk./.usageGB‚287.275447845459 disk.sda2.out‚0( disk.nvme0c0n1.out‚72884.22607421875 disk.dm-0.in‚ 471.90234375 disk.drbd2266.out‚0 disk.rbd1.out‚ 8.4160156253M Í’ƒ)‚(:‡ —ÐåÔÒœµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚286.885' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.377 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{„xóp„)‚(:å —ÐåÔ€Õ¶ disk.nvme0n1.in‚ 471.90234375 cpu‚25.274728613764335 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 393.21875 proc.cpu.threads‚17" disk.dm-0.out‚72925.9208984375 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚72925.9208984375 disk.dm-1.in‚0.203125( proc.memory.availableMB‚ 332.12890625 network.sent‚ 2.6768574e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 8.572265625" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.in‚0 network.recv‚ 3.535172e+06 disk.drbd2266.out‚0 disk.dm-0.in‚ 471.90234375 disk.sda.in‚0.203125 disk.sda2.in‚0! proc.memory.rssMB‚ 2952.328125% disk./.usageGB‚287.29096603393555 disk.sda3.out‚ 393.21875 disk.rbd0.out‚0* proc.memory.percent‚18.019580841064453 disk.rbd0.in‚0 disk.dm-1.out‚ 393.21875$ memory_percent‚97.97284603118896* disk./.usagePercent‚19.280337783713694' disk.nvme0c0n1.out‚72925.9208984375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0H"§…)‚(:’ ¦ÐåÔ‚û”µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç\—x†)‚(:í ¦ÐåÔžÈÚ¶% disk.nvme0n1.out‚72966.1630859375" disk.nvme0c0n1.in‚ 471.90234375$ disk./.usageGB‚287.2886085510254 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 8.572265625 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125# proc.memory.rssMB‚ 2952.33203125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 393.73046875 disk.drbd2265.in‚0 disk.dm-1.out‚ 393.73046875 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚18.019604682922363 network.recv‚ 3.538613e+06 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.sda3.out‚ 393.73046875$ memory_percent‚97.97282218933105 cpu‚25.274409487825523 network.sent‚ 2.6773563e+07 disk.rbd0.in‚0" disk.dm-0.out‚72966.1630859375) disk./.usagePercent‚19.28017957105753 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚72966.1630859375' proc.memory.availableMB‚ 332.1328125À²¼4‡)/ _wandbruntime‚10550  _runtime‚10550”<´Õʈ)Ä ' lr-AdamW/pg1‚1.0603156538793626e-05 ' lr-AdamW/pg2‚1.0603156538793626e-05 trainer/global_step‚10499 % _timestamp‚1.7884385725254014e+09 _runtime‚10550.88048203 _step‚860ƒ'u—ω)É ' lr-AdamW/pg1‚1.0603156538793626e-05 ' lr-AdamW/pg2‚1.0603156538793626e-05 trainer/global_step‚10499 % _timestamp‚1.7884385725254014e+09 _runtime‚10550.88048203 _step‚860Ü@q4Š)/ _wandbruntime‚10550  _runtime‚10550P‚@¯"(‹)œP F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112531137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226057078 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226033972 # _timestamp‚1.78843857252613e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 188142903 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22431568742912 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188142909 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194966073 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1401523636827136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1425884077075456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112531140 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1448318751212032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22431568742912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82434933 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 414199987 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22396742642177 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22434674136576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1425884050683904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22434674136576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22393639173253 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188142903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226057078 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82434393 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1249422295085568 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1218312147819008 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1218312119684608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1448315619426816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1425884050683904 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31110230415360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1401523611399168 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1448315619426816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 414176875 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226033972 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 188142909 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1425884077075456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1423920379469313 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1423917250572421 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1448315619426816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1425884050683904 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31110230415360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1401523611399168 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1448315619426816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 414176875 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226033972 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 188142909 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1425884077075456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1423920379469313 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1423917250572421 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194966073 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1401523636827136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1425884077075456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112531140 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1448318751212032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22431568742912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82434933 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 414199987 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22396742642177 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22434674136576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1425884050683904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22434674136576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22393639173253 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302Ýrc‡ü4)/ _wandbruntime‚10551  _runtime‚10551¨Åþe¶Ž)¨N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1448452357627392 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 414215249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚10499 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1448452357627392 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1448449225828864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22436786463744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188160187 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22398851435485 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226078194 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194983928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1218424558683136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1249537724919296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YFDevi¦û©xceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22395747966353 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1426015544772096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22436786463744 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194983418 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 414215249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22433681056768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1426015571163648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1426015544772096 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31113166236160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚10551.777638283 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82442535 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1218424530548736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112541390 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414238387 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1401653033828352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226055062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚862 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414238387 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1401653033828352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226055062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚862 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1218424558683136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1249537724919296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22395747966353 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1426015544772096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22436786463744 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194983418 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 414215249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22433681056768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1426015571163648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1426015544772096 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31113166236160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚10551.777638283 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82442535Þ ÐZ4)/ _wandbruntime‚10551  _runtime‚10551Y€ÿ/c‘)Ý 2loss/SRqqt1/assignment_loss‚0.8292458620545151 1loss/SRqqt2/detection_loss‚2.4638803005218506 &loss/total_loss‚39.429362735716836 3loss/FBt1/assignment_loss‚0.0020643024606050884 .loss/FRt3/detection_loss‚2.162500673014185 /loss/FBt2/detection_loss‚2.5190156579017637 trainer/global_step‚10499 _runtime‚10551.778740053 2loss/FBt2/assignment_loss‚0.014942965668160469 % _timestamp‚1.7884385734237845e+09 0loss/FRt2/assignment_loss‚2.6856100779992564 2loss/SRqqt4/assignment_loss‚1.4862406984902918 /loss/FRt2/detection_loss‚2.2740770313474883 1loss/SRqqt1/detection_loss‚0.8977254233715763 /loss/SRqqt4/detection_loss‚2.05168452039361 /loss/FBt1/detection_loss‚0.8344861432187148 2loss/SRqqt2/assignment_loss‚1.3916482241558177 2loss/FBt3/assignment_loss‚0.010149864390119287 1loss/FBt4/assignment_loss‚0.04170511131731193 .loss/FRt1/detection_loss‚1.483660486108367 .loss/FRt4/detection_loss‚2.870078690590397 .loss/FBt3/detection_loss‚2.882418828351157 epoch‚8 _step‚863 0loss/FRt1/assignment_loss‚1.0835623067358267 1loss/SRqqt3/detection_loss‚1.4852727984945322 /loss/FBt4/detection_loss‚2.0442927456819096 /loss/FRt3/assignment_loss‚3.508723321168319 0loss/FRt4/assignment_loss‚2.9575242457851285 2loss/SRqqt3/assignment_loss‚1.4488524564959366X«Nh’)â % _timestamp‚1.7884385734237845e+09 0loss/FRt2/assignment_loss‚2.6856100779992564 2loss/SRqqt4/assignment_loss‚1.4862406984902918 /loss/FRt2/detection_loss‚2.2740770313474883 1loss/SRqqt1/detection_loss‚0.8977254233715763 /loss/SRqqt4/detection_loss‚2.05168452039361 /loss/FBt1/detection_loss‚0.8344861432187148 2loss/SRqqt2/assignment_loss‚1.3916482241558177 2loss/FBt3/assignment_loss‚0.010149864390119287 1loss/FBt4/assignment_loss‚0.04170511131731193 .loss/FRt1/detection_loss‚1.483660486108367 .loss/FRt4/detection_loss‚2.870078690590397 .loss/FBt3/detection_loss‚2.882418828351157 epoch‚8 _step‚863 0loss/FRt1/assignment_loss‚1.0835623067358267 1loss/SRqqt3/detection_loss‚1.4852727984945322 /loss/FBt4/detection_loss‚2.0442927456819096 /loss/FRt3/assignment_loss‚3.508723321168319 0loss/FRt4/assignment_loss‚2.9575242457851285 2loss/SRqqt3/assignment_loss‚1.4488524564959366 2loss/SRqqt1/assignment_loss‚0.8292458620545151 1loss/SRqqt2/detection_loss‚2.4638803005218506 &loss/total_loss‚39.429362735716836 3loss/FBt1/assignment_loss‚0.0020643024606050884 .loss/FRt3/detection_loss‚2.162500673014185 /loss/FBt2/detection_loss‚2.5190156579017637 trainer/global_step‚10499 _runtime‚10551.778740053 2loss/FBt2/assignment_loss‚0.014942965668160469ß-¦sà“)‚(:’ µÐåÔÝ™Œµ gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚279.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ð|”)‚(:ñ µÐåÔèÄø¶ disk.drbd2266.out‚0$ memory_percent‚97.97453880310059 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.drbd2266.in‚0* disk./.usagePercent‚19.282438069524602% disk./.usageGB‚287.32226181030273 disk.sda3.out‚ 394.40234375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd1.out‚ 8.7841796875" disk.nvme0c0n1.in‚ 471.90234375' proc.memory.availableMB‚ 331.8515625 network.recv‚ 3.543585e+06& disk.nvme0n1.out‚73026.31298828125( disk.nvme0c0n1.out‚73026.31298828125 network.sent‚ 2.6779899e+07 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2952.33203125 disk.dm-0.in‚ 471.90234375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 394.40234375* proc.memory.percent‚18.019604682922363 cpu‚25.2740667172456# disk.dm-0.out‚73026.31298828125 disk.sda.out‚ 394.40234375 disk.sda.in‚0.203125»†åÊ•)‚(:’ ÄÐåÔÊÉø¶ gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.535' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.30700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦‹|DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1255189407427072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10597.833028541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1430495682776321 trainer/global_step‚10549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1255189490575872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1432458755093504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454995953943040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22540304243200 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454995953943040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31256744536064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1430492553879429 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454999085728256 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195860343 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195859796 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1223932691025408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189007109 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1407993486887936 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227091824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454999085728256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454995953943040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31256744536064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1430492553879429 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454999085728256 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195860343 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195859796 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1223932691025408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189007109 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1407993486887936 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227091824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454999085728256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1255189407427072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10597.833028541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1430495682776321 trainer/global_step‚10549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1255189490575872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1432458755093504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454995953943040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0á®KÜ,4Ÿ)/ _wandbruntime‚10598  _runtime‚10598¤ÎjK,' )¦N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416160439 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195878101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1430624059673745 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚10598.71876872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1432590275573248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22501201212561 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227136046 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189024387 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1455132692143616 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416160439 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227136046 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113054043 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1408122883889152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1432590249181696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22504304681693 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189024393 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416137301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227112914 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31259658743808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1455132692143616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416137301 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1455129560345088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227112914 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195877564 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82823524 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1408122858461184 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302]»~Q1'¡)«N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113054040 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1432590275573248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22542416570368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚10598.71876872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1432590275573248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22501201212561 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227136046 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189024387 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1455132692143616 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416160439 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227136046 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113054043 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1408122883889152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1432590249181696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22504304681693 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189024393 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416137301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227112914 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31259658743808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1455132692143616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416137301 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1455129560345088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227112914 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195877564 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82823524 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1408122858461184 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416160439 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195878101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1430624059673745 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189871303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1414463336948736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113556437 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22642828956160 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1229553234231808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1461676288459264 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196752565 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31403353051648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228172782 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1437067857186437 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113556440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1461676288459264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22645934349824 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189871309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0úŸ1W)(°)£P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1439033485894656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22604545665669 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1260956559149056 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1229553206097408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚869 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1461679420244480 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1260956642297856 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884386666763687e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1439033485894656 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22607649134593 trainer/global_step‚10599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228172782 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1439033459503104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83196128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189871309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228149676 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1461679420244480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418044091 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189871303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1414463336948736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113556437 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22642828956160 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1229553234231808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1461676288459264 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196752565 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31403353051648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228172782 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1437067857186437 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113556440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1461676288459264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22645934349824 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189871309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196753109 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _runtime‚10645.031598416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189871303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31403408066048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418020979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22645934349824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1439033459503104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22642828956160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228149676 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83196669 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1229665616961536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31406261825536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22609757927901 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1461813026659840 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228170766 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189888593 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189888587 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1439164953591296 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228193898 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1461813026659840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418082491 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189888587 T?¦ÍìF˜DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1437199362980753 _step‚870 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22648046676992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83204015 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31406316826624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22606654458769 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1261071878787072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418059353 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1414592708521984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚10645.927528165 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196770189 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228193898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1261071961922560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418082491 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1461809894861312 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1439164979982848 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196770708 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113566693 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0%_q3'³)­N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _runtime‚10645.927528165 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196770189 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228193898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1261071961922560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418082491 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1461809894861312 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1439164979982848 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196770708 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113566693 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1229665616961536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31406261825536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22609757927901 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1461813026659840 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228170766 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189888593 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189888587 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1439164953591296 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228193898 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1461813026659840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418082491 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189888587 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1437199362980753 _step‚870 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22648046676992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83204015 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31406316826624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22606654458769 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1261071878787072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418059353 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1414592708521984ævã€4´)/ _wandbruntime‚10645  _runtime‚10645jyËedµ)Þ 1loss/SRqqt4/detection_loss‚2.1635481075328937 %loss/total_loss‚38.43993347145098 _runtime‚10645.92901935 1loss/FBt4/assignment_loss‚0.10658448971876436 1loss/SRqqt3/detection_loss‚1.6896626392548735 trainer/global_step‚10599 _step‚871 /loss/FRt3/assignment_loss‚2.012072558586414 2loss/FBt1/assignment_loss‚0.031721492811785955 /loss/FRt2/detection_loss‚2.7036864396772886 /loss/FRt4/detection_loss‚2.6375568636826108 % _timestamp‚1.7884386675737357e+09 0loss/FRt1/assignment_loss‚1.3116326544966017 /loss/FBt1/detection_loss‚0.6157443247983853 2loss/SRqqt2/assignment_loss‚1.6496733501036134 /loss/FRt1/detection_loss‚1.4771197291357177 /loss/FRt3/detection_loss‚2.2654666121189413 1loss/SRqqt2/detection_loss‚2.0890995264053345 0loss/FRt2/assignment_loss‚2.9732609234358134 0loss/FRt4/assignment_loss‚3.3481138808386666 2loss/SRqqt4/assignment_loss‚1.2490492808249067 /loss/FBt3/detection_loss‚1.9950176018934984 1loss/FBt2/assignment_loss‚0.22567641246132553 /loss/FBt2/detection_loss‚2.8747320249676704 /loss/FBt4/detection_loss‚1.5379113256931305 1loss/FBt3/assignment_loss‚0.19180428223276416 epoch‚8 2loss/SRqqt1/assignment_loss‚1.0371785561617373 2loss/SRqqt3/assignment_loss‚1.3293236445635557 1loss/SRqqt1/detection_loss‚0.9242967500546833©YÀDi¶)ã 0loss/FRt1/assignment_loss‚1.3116326544966017 /loss/FBt1/detection_loss‚0.6157443247983853 2loss/SRqqt2/assignment_loss‚1.6496733501036134 /loss/FRt1/detection_loss‚1.4771197291357177 /loss/FRt3/detection_loss‚2.2654666121189413 1loss/SRqqt2/detection_loss‚2.0890995264053345 0loss/FRt2/assignment_loss‚2.9732609234358134 0loss/FRt4/assignment_loss‚3.3481138808386666 2loss/SRqqt4/assignment_loss‚1.2490492808249067 /loss/FBt3/detection_loss‚1.9950176018934984 1loss/FBt2/assignment_loss‚0.22567641246132553 /loss/FBt2/detection_loss‚2.8747320249676704 /loss/FBt4/detection_loss‚1.5379113256931305 1loss/FBt3/assignment_loss‚0.19180428223276416 epoch‚8 2loss/SRqqt1/assignment_loss‚1.0371785561617373 2loss/SRqqt3/assignment_loss‚1.3293236445635557 1loss/SRqqt1/detection_loss‚0.9242967500546833 1loss/SRqqt4/detection_loss‚2.1635481075328937 %loss/total_loss‚38.43993347145098 _runtime‚10645.92901935 1loss/FBt4/assignment_loss‚0.10658448971876436 1loss/SRqqt3/detection_loss‚1.6896626392548735 trainer/global_step‚10599 _step‚871 /loss/FRt3/assignment_loss‚2.012072558586414 2loss/FBt1/assignment_loss‚0.031721492811785955 /loss/FRt2/detection_loss‚2.7036864396772886 /loss/FRt4/detection_loss‚2.6375568636826108 % _timestamp‚1.7884386675737357e+09ç:9¡ž·)‚(:“ ÑåÔª¿Áµ gpu.0.gpu‚80.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚209.487' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.897400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É^²·v¸)‚(:ë ÑåÔùД·# proc.memory.rssMB‚ 2952.33203125 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 397.5234375 disk.rbd1.in‚0 cpu‚25.272153254496374 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda1.out‚0$ memory_percent‚97.97210693359375 disk.drbd2266.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚73413.56298828125* proc.memory.percent‚18.019604682922363* disk./.usagePercent‚19.289968787151814% disk./.usageGB‚287.43447494506836 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.sda3.out‚ 397.5234375 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 471.90234375& disk.nvme0n1.out‚73413.56298828125 disk.sda2.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 9.2529296875# disk.dm-0.out‚73413.56298828125" proc.memory.availableMB‚332.25 network.sent‚ 2.7020677e+07 network.recv‚ 3.572441e+06 disk.sda.out‚ 397.5234375 disk.sda1.in‚0 disk.drbd1716.in‚0zW U¹)‚(:’ žÑåÔùÒ›µ gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚298.078' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.61559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K ŒXvº)‚(:ë žÑåԋȶ disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚97.97210693359375 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375& disk.nvme0n1.out‚73473.70361328125 disk.drbd1716.out‚0" proc.memory.availableMB‚332.25 network.sent‚ 2.7129388e+07 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 397.90234375 disk.rbd1.out‚ 9.2529296875 disk.drbd2080.in‚0 disk.sda.out‚ 397.90234375 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 471.90234375$ disk./.usageGB‚287.4626922607422( disk.nvme0c0n1.out‚73473.70361328125# proc.memory.rssMB‚ 2952.33203125 disk.drbd2265.in‚0 disk.drbd2266.out‚0# disk.dm-0.out‚73473.70361328125 disk.sda3.in‚0.203125 disk.sda3.out‚ 397.90234375 network.recv‚ 3.57868e+06) disk./.usagePercent‚19.29186247488259 disk.drbd2266.in‚0* proc.memory.percent‚18.019604682922363 cpu‚25.271821140427154" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.out‚0±Ì3—ž»)‚(:“ ­ÑåÔàªû´ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.451' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.690200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªšî§r¼)‚(:ç ­ÑåÔ°íª¶ disk.dm-0.out‚73548.94921875$ memory_percent‚97.97365665435791# proc.memory.rssMB‚ 2952.33203125) disk./.usagePercent‚19.29245590034696 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375$ disk./.usageGB‚287.4715347290039 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚18.019604682922363 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.sda3.out‚ 398.70703125 disk.rbd0.in‚0# disk.nvme0n1.out‚73548.94921875 disk.dm-1.out‚ 398.70703125 disk.rbd1.in‚0% disk.nvme0c0n1.out‚73548.94921875 disk.sda1.in‚0 disk.drbd1716.out‚0 cpu‚25.271488102133212 disk.drbd2080.out‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125( proc.memory.availableMB‚ 331.99609375 network.sent‚ 2.7135743e+07 network.recv‚ 3.583328e+06 disk.sda.out‚ 398.70703125 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 9.28515625 disk.dm-0.in‚ 471.90234375NÏx4½)/ _wandbruntime‚10692  _runtime‚10692Ľi:ɾ)à trainer/global_step‚10649 % _timestamp‚1.7884387140398214e+09 _runtime‚10692.394833887 _step‚872 & lr-AdamW/pg1‚8.725680107846112e-06 & lr-AdamW/pg2‚8.725680107846112e-06´'pXο)È & lr-AdamW/pg1‚8.725680107846112e-06 & lr-AdamW/pg2‚8.725680107846112e-06 trainer/global_step‚10649 % _timestamp‚1.7884387140398214e+09 _runtime‚10692.394833887 _step‚872è/xÑÇ4À)/ _wandbruntime‚10692  _runtime‚10692§LûŽ#Á)žP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22748459062784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190735509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1445608190304256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1443646289390337 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22751564456448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884387140405757e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197644999 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114069090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 419966143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1420933187009536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190735503 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229207528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1445608190304256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468359754760704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31549842363392 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83575912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468356622975488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1445608163912704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229230634 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1235173749303808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22713102380801 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22709998911877 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31549787348992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229207528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190735509 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1445608163912704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22748459062784 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1266723619801600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114069087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22751564456448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21pã¶ê–880356864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚873 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197645553 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468359754760704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1420933161581568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468356622975488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83576463 _runtime‚10692.395881263 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229230634 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1235173777438208 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878ŠgøT)(Â)£P BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1445608163912704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22748459062784 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1266723619801600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114069087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22751564456448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _step‚873 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197645553 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468359754760704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1420933161581568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468356622975488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83576463 _runtime‚10692.395881263 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229230634 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1235173777438208 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22748459062784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190735509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1445608190304256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1443646289390337 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22751564456448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884387140405757e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197644999 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114069090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 419966143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1420933187009536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190735503 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229207528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1445608190304256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468359754760704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31549842363392 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83575912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468356622975488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1445608163912704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229230634 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1235173749303808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22713102380801 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22709998911877 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31549787348992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229207528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190735509éÓÙ>ª4Ã)/ _wandbruntime‚10693  _runtime‚10693χ^.'Ä)¨N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197662674 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚10693.302092322 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1235286188302336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31552697624064 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1468493361176064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1468490229377536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22712107704977 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468493361176064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229251750 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 419981405 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229228618 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22753676783616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229228618 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22753676783616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229251750 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114079343 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22750571376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83583870 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190752793 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1443774666287761 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1443777795184861 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190752787 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1445739684392448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22715211174109 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1445739684392448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420004543 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83583334 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1445739658000896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1266838857792000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884387149461849e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197663213 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1235286160167936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 419981405 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31552754722304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190752787 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 _step‚874 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240ZÉ23'Å)­N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229228618 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22753676783616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229228618 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22753676783616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229251750 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114079343 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22750571376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83583870 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190752793 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1443774666287761 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1443777795184861 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190752787 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1445739684392448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22715211174109 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1445739684392448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420004543 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83583334 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1445739658000896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1266838857792000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884387149461849e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197663213 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1235286160167936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 419981405 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31552754722304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190752787 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 _step‚874 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197662674 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚10693.302092322 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1235286188302336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31552697624064 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1468493361176064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1468490229377536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22712107704977 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468493361176064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229251750 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 419981405 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440êŽèþ4Æ)/ _wandbruntime‚10693  _runtime‚10693ˆyÅ%GÇ)Ý 0loss/FRt4/assignment_loss‚3.4773566325505576 2loss/SRqqt1/assignment_loss‚0.9998572294986466 3loss/FBt1/assignment_loss‚0.0005006324735720979 1loss/SRqqt3/detection_loss‚1.4823946777665824 /loss/FBt2/detection_loss‚2.6713858942190805 _runtime‚10693.303221514 1loss/FBt3/assignment_loss‚0.05274588178478249 .loss/FRt1/detection_loss‚1.547781117542675 0loss/FBt1/detection_loss‚0.37179717952655494 2loss/SRqqt2/assignment_loss‚1.3783177056390306 1loss/SRqqt1/detection_loss‚0.9358262082152977 1loss/SRqqt2/detection_loss‚2.9008387845495474 /loss/FBt3/detection_loss‚2.0341082334518434 0loss/FRt1/assignment_loss‚1.6009316742420197 /loss/FRt2/detection_loss‚2.3439420920771523 epoch‚8 % _timestamp‚1.7884387149482548e+09 2loss/SRqqt3/assignment_loss‚1.2322754495238002 1loss/SRqqt4/detection_loss‚2.5470953473338374 /loss/FRt3/assignment_loss‚2.497434530772415 2loss/SRqqt4/assignment_loss‚1.5036758401051715 0loss/FBt4/assignment_loss‚0.3522144264752569 .loss/FBt4/detection_loss‚2.77212473005056’ž¬É4 %loss/total_loss‚41.33280567339945 _step‚875 1loss/FBt2/assignment_loss‚0.33411350989869487 /loss/FRt3/detection_loss‚1.9951278862415576 /loss/FRt4/detection_loss‚3.0585423420976707 trainer/global_step‚10649 /loss/FRt2/assignment_loss‚3.242417667363141̳àähÈ)â trainer/global_step‚10649 /loss/FRt2/assignment_loss‚3.242417667363141 0loss/FRt4/assignment_loss‚3.4773566325505576 2loss/SRqqt1/assignment_loss‚0.9998572294986466 3loss/FBt1/assignment_loss‚0.0005006324735720979 1loss/SRqqt3/detection_loss‚1.4823946777665824 /loss/FBt2/detection_loss‚2.6713858942190805 _runtime‚10693.303221514 1loss/FBt3/assignment_loss‚0.05274588178478249 .loss/FRt1/detection_loss‚1.547781117542675 0loss/FBt1/detection_loss‚0.37179717952655494 2loss/SRqqt2/assignment_loss‚1.3783177056390306 1loss/SRqqt1/detection_loss‚0.9358262082152977 1loss/SRqqt2/detection_loss‚2.9008387845495474 /loss/FBt3/detection_loss‚2.0341082334518434 0loss/FRt1/assignment_loss‚1.6009316742420197 /loss/FRt2/detection_loss‚2.3439420920771523 epoch‚8 % _timestamp‚1.7884387149482548e+09 2loss/SRqqt3/assignment_loss‚1.2322754495238002 1loss/SRqqt4/detection_loss‚2.5470953473338374 /loss/FRt3/assignment_loss‚2.497434530772415 2loss/SRqqt4/assignment_loss‚1.5036758401051715 0loss/FBt4/assignment_loss‚0.3522144264752569 .loss/FBt4/detection_loss‚2.772124730050564 %loss/total_loss‚41.33280567339945 _step‚875 1loss/FBt2/assignment_loss‚0.33411350989869487 /loss/FRt3/detection_loss‚1.9951278862415576 /loss/FRt4/detection_loss‚3.0585423420976707ëºZº¾žÉ)‚(:“ ¼ÑåÔ½™¶¶ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚279.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.975199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}{(²bÊ)‚(:× ¼ÑåÔöÃи network.recv‚ 3.586798e+06 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 73599.0859375 disk.dm-0.out‚ 73599.0859375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 332.18359375 disk.sda3.in‚0.203125 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2952.33203125 cpu‚25.271190268086706 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 9.28515625$ memory_percent‚97.97251224517822 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚18.019604682922363 proc.cpu.threads‚17 network.sent‚ 2.7141032e+07 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.out‚399.0625" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.out‚0) disk./.usagePercent‚19.29348530664214# disk./.usageGB‚287.486873626709 disk.drbd2266.out‚0 disk.sda3.out‚399.0625 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.sda.out‚399.0625" disk.nvme0n1.out‚ 73599.0859375÷Ä%ÞžË)‚(:“ ËÑåÔʽ¢µ gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚259.478' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.895599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:xzKsÌ)‚(:è ËÑåÔú§Ù¶ disk.drbd2080.out‚0 disk.nvme0n1.in‚ 471.90234375( proc.memory.availableMB‚ 332.18359375 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.270867399855344 network.sent‚ 2.7250017e+07" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.out‚ 399.5859375 disk.sda.out‚ 399.5859375 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 399.5859375$ memory_percent‚97.97251224517822 disk.sda2.in‚0 disk.drbd2266.in‚0 network.recv‚ 3.593945e+06 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 9.44140625* proc.memory.percent‚18.019604682922363$ disk./.usageGB‚287.4831771850586 disk.sda1.in‚0$ disk.nvme0n1.out‚73662.216796875* disk./.usagePercent‚19.293237235341458& disk.nvme0c0n1.out‚73662.216796875! disk.dm-0.out‚73662.216796875 disk.sda.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2952.33203125 proc.cpu.threads‚17 disk.sda3.in‚0.203125›Ôœ’Í)‚(:‡ ÚÑåÔ²‡‰µ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚257.591' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5182 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rw5¹wÎ)‚(:ì ÚÑåÔ·´ç· proc.cpu.threads‚17 network.recv‚ 3.597861e+06 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 399.9921875" disk.dm-0.out‚73711.4130859375( proc.memory.availableMB‚ 332.18359375* disk./.usagePercent‚19.294341907851706 disk.sda2.out‚0' disk.nvme0c0n1.out‚73711.4130859375 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0 network.sent‚ 2.7255876e+07 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.sda.out‚ 399.9921875 disk.sda2.in‚0 disk.sda3.out‚ 399.9921875# proc.memory.rssMB‚ 2952.33203125 disk.dm-1.in‚0.203125 disk.sda.in‚0.203125 disk.rbd1.out‚ 9.44140625$ memory_percent‚97.97251224517822 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 471.90234375% disk.nvme0n1.out‚73711.4130859375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0% disk./.usageGB‚287.49963760375977 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2080.out‚0* proc.memory.percent‚18.019604682922363 cpu‚25.270553829805127<þË4Ï)/ _wandbruntime‚10739  _runtime‚10739e¼QeÉÐ)à & lr-AdamW/pg1‚8.138680093913095e-06 & lr-AdamW/pg2‚8.138680093913095e-06 trainer/global_step‚10699 % _timestamp‚1.7884387613214765e+09 _runtime‚10739.676531213 _step‚876[÷î¡ÎÑ)È & lr-AdamW/pg1‚8.138680093913095e-06 & lr-AdamW/pg2‚8.138680093913095e-06 trainer/global_step‚10699 % _timestamp‚1.7884387613214765e+09 _runtime‚10739.676531213 _step‚876ìGMZ£4Ò)/ _wandbruntime‚10739  _runtime‚10739]ÈŽ®#(Ó)P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191599703 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198536939 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 191599709 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1240794292510208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114581740 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 421865083 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114581737 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22818555627009 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1452182868322304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1452182894713856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83955760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _step‚877 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 421888195 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31695847287808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22815452158085 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230288486 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22854089169408 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1450221592697345 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1450218463800453 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1427403011642368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1475040089276928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1452182894713856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1272490222946816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1475036957491712 trainer/global_step‚10699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31695902302208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1475040089276928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230288486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 _runtime‚10739.67758384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1475036957491712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 191599703 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22857194563072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1452182868322304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230265380 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83955202 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230265380 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400NMØ((Ô)¢P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 191599709 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1240794292510208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114581740 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 421865083 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114581737 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22818555627009 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1452182868322304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1452182894713856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83955760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 _step‚877 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 421888195 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31695847287808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22815452158085 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230288486 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22854089169408 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1450221592697345 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1450218463800453 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1427403011642368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1475040089276928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1452182894713856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1272490222946816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1475036957491712 trainer/global_step‚10699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31695902302208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1475040089276928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230288486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 _runtime‚10739.67758384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1475036957491712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 191599703 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22857194563072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1452182868322304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230265380 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83955202 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230265380 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191599703 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198536939 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249í'Ê~ôžÕ)‚(:“ éÑåÔÂî´µ gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.839200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“}v:yÖ)‚(:î éÑåÔÖ¾ˆ· disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.in‚0# memory_percent‚97.9724645614624 disk.rbd1.out‚ 9.4736328125 network.recv‚ 3.602445e+06 disk.sda3.in‚0.203125( proc.memory.availableMB‚ 332.19140625 cpu‚25.27025011454922 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0% disk.nvme0n1.out‚73788.8154296875* proc.memory.percent‚18.019604682922363* disk./.usagePercent‚19.293023469050368 disk.sda.out‚ 400.65234375 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2952.33203125 disk.sda2.out‚0 disk.dm-1.out‚ 400.65234375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 network.sent‚ 2.7262285e+07$ disk./.usageGB‚287.4799919128418 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚73788.8154296875 disk.sda2.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.out‚ 400.65234375 disk.drbd2080.in‚0" disk.dm-0.out‚73788.8154296875Ñæ4×)/ _wandbruntime‚10740  _runtime‚10740mMǾOØ)¨N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1450353098491869 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198555252 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22856201483264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10699 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191616987 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1452314362410496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 421903457 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191616987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1427532408643584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191616993 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1272605554623488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31698823114752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1240906703374336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198554696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1427532434071552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22856201483264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31698766016512 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230286470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1452314388802048 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114591990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884387622141564e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚10740.570169662 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114591993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83963259 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22820664420317 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1475173695692288 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1452314362410496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1475173695692288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83962706 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1450349969594769 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 421903457 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 421926595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230286470 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 230309602 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984FÅÝV3'Ù)­N YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1272605469390848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚878 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22859306890240 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22817560951185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1240906731508736 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 421903457 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191616987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1427532408643584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191616993 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1272605554623488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31698823114752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1240906703374336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198554696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1427532434071552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22856201483264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31698766016512 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230286470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1452314388802048 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114591990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884387622141564e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _runtime‚10740.570169662 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114591993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83963259 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22820664420317 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1475173695692288 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1452314362410496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1475173695692288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83962706 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1450349969594769 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 421903457 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 421926595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230286470 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 230309602 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1450353098491869 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198555252 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22856201483264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10699 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191616987 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1452314362410496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1481717292007936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231346338 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115094390 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1278256480109056 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31841699406848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 423810247 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192463909 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22924008873217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22962824669696 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884388084031231e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1246414863851008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1456793767107461 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192463903 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1433872887131136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231323232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1458757599123456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199429590 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1456796896004353 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31841644392448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚10786.75838438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1458757599123456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84334649 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 423787135 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199429036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1458757572731904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231323232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMoniD€­Î˜tor.on_train_batch_start/requested_bytes.large_pool.freed‚1433872861703168 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22920905404293 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1458757572731904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22959719276032 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1246414835716608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚10749 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192463903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1456796896004353 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚31841644392448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _runtime‚10786.75838438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1458757599123456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84334649 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 423787135 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199429036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1458757572731904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231323232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1433872861703168 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22920905404293 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1458757572731904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22959719276032 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1246414835716608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚10749 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192463903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1481717292007936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231346338 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115094390 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1278256480109056 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31841699406848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 423810247 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192463909 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22924008873217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22962824669696 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216ñ_­Ž¨4é)/ _wandbruntime‚10787  _runtime‚107878VR.'ê)¨N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1458889093211648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884388092917018e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1246527274715136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1434002258704384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231367454 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚882 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 423848647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1278371919834112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1434002284132352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199446878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192481187 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22964936996864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22961831589888 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115104643 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1456925272901777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192481193 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1456928401798877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1481854030208512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 423848647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231367454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84342770 _runtime‚10787.647468214 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1458889066820096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199447413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0»!! 3'ë)­N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1278371919834112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1434002284132352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199446878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192481187 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22964936996864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22961831589888 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115104643 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1456925272901777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192481193 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1456928401798877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1481854030208512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 423848647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231367454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84342770 _runtime‚10787.647468214 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1458889066820096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199447413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1458889093211648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884388092917018e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1246527274715136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1434002258704384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231367454 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚882 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 423848647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180òœ©ü4ì)/ _wandbruntime‚10787  _runtime‚10787©óBöFí)Û .loss/FBt3/detection_loss‚2.154808759689331 .loss/FBt4/detection_loss‚2.216028183698654 %loss/total_loss‚38.79960751853995 2loss/SRqqt4/assignment_loss‚1.4410891449910455 /loss/FRt1/detection_loss‚1.2244861953714419 .loss/FRt3/detection_loss‚2.079073821504911 0loss/SRqqt2/detection_loss‚2.6T`Y17892937734723 epoch‚8 trainer/global_step‚10749 0loss/FRt4/assignment_loss‚3.1472254196802774 4loss/FBt1/assignment_loss‚0.00018432624988182537 /loss/FRt4/detection_loss‚2.9458259615031155 1loss/SRqqt1/detection_loss‚0.7246917131478372 /loss/FRt2/assignment_loss‚2.943273348193015 1loss/SRqqt2/assignment_loss‚1.079372685868293 0loss/FBt3/assignment_loss‚0.1581129246559902 1loss/SRqqt3/detection_loss‚1.7256683358606302 /loss/FBt1/detection_loss‚0.5161615276539867 0loss/FRt1/assignment_loss‚1.0865417151224046 0loss/FRt3/assignment_loss‚2.6936343752202534 2loss/SRqqt3/assignment_loss‚1.4985287467123203 1loss/SRqqt4/detection_loss‚2.6474713597978865 % _timestamp‚1.7884388092935312e+09 _runtime‚10787.648552771 0loss/FBt2/assignment_loss‚0.4124141804873943 2loss/SRqqt1/assignment_loss‚0.7688371850678738 2loss/FBt4/assignment_loss‚0.028599193348782137 /loss/FBt2/detection_loss‚1.7955661594867707 /loss/FRt2/detection_loss‚2.8941193174931312 _step‚883årfî)à /loss/FRt2/detection_loss‚2.8941193174931312 _step‚883 .loss/FBt3/detection_loss‚2.154808759689331 .loss/FBt4/detection_loss‚2.216028183698654 %loss/total_loss‚38.79960751853995 2loss/SRqqt4/assignment_loss‚1.4410891449910455 /loss/FRt1/detection_loss‚1.2244861953714419 .loss/FRt3/detection_loss‚2.079073821504911 0loss/SRqqt2/detection_loss‚2.617892937734723 epoch‚8 trainer/global_step‚10749 0loss/FRt4/assignment_loss‚3.1472254196802774 4loss/FBt1/assignment_loss‚0.00018432624988182537 /loss/FRt4/detection_loss‚2.9458259615031155 1loss/SRqqt1/detection_loss‚0.7246917131478372 /loss/FRt2/assignment_loss‚2.943273348193015 1loss/SRqqt2/assignment_loss‚1.079372685868293 0loss/FBt3/assignment_loss‚0.1581129246559902 1loss/SRqqt3/detection_loss‚1.7256683358606302 /loss/FBt1/detection_loss‚0.5161615276539867 0loss/FRt1/assignment_loss‚1.0865417151224046 0loss/FRt3/assignment_loss‚2.6936343752202534 2loss/SRqqt3/assignment_loss‚1.4985287467123203 1loss/SRqqt4/detection_loss‚2.6474713597978865 % _timestamp‚1.7884388092935312e+09 _runtime‚10787.648552771 0loss/FBt2/assignment_loss‚0.4124141804873943 2loss/SRqqt1/assignment_loss‚0.7688371850678738 2loss/FBt4/assignment_loss‚0.028599193348782137 /loss/FBt2/detection_loss‚1.7955661594867707ó1ô’ï)‚(:‡ ¥ÒåԽʟµ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚290.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.133 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É}^Azð)‚(:ï ¥ÒåÔòÝü¶ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 402.76171875 disk.drbd2265.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚74019.2939453125 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125# proc.memory.rssMB‚ 2952.33203125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚74019.2939453125 disk.dm-0.in‚ 471.90234375 disk.dm-1.out‚ 402.76171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 330.75390625 network.recv‚ 3.629645e+06* disk./.usagePercent‚19.300914901293325 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0* proc.memory.percent‚18.019604682922363 cpu‚25.269006751737145 disk.nvme0n1.in‚ 471.90234375 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125$ memory_percent‚97.98123836517334 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd1.out‚ 9.662109375" disk.dm-0.out‚74019.2939453125 disk.drbd2080.in‚0 disk.sda.out‚ 402.76171875 network.sent‚ 2.7473123e+07$ disk./.usageGB‚287.59757995605475 DNžñ)‚(:“ ´ÒåÔ¯Äâµ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.084' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.616800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µTu”zò)‚(:ï ´ÒåÔ‚ù‰· disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.out‚0 network.sent‚ 2.7581325e+07 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.drbd2265.in‚0 cpu‚25.26868015663375 disk.rbd0.in‚0% disk./.usageGB‚287.62793350219727 disk.rbd1.out‚ 9.662109375 disk.sda3.out‚ 403.30078125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚0.203125* disk./.usagePercent‚19.302951953243273 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 403.30078125$ memory_percent‚97.97995090484619' disk.nvme0c0n1.out‚74076.0048828125 disk.sda1.out‚0# proc.memory.rssMB‚ 2952.33203125 proc.cpu.threads‚17" disk.dm-0.out‚74076.0048828125 disk.drbd2265.out‚0% disk.nvme0n1.out‚74076.0048828125 disk.sda2.out‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 330.96484375* proc.memory.percent‚18.019604682922363 disk.sda.out‚ 403.30078125 disk.rbd1.in‚0 network.recv‚ 3.635509e+06ÒñØó)‚(:’ ÃÒåÔÔ‹ µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚286.239' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.24779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nvô)‚(:ë ÃÒåÔ††ü¶ disk.drbd1716.out‚0 network.recv‚ 3.640105e+06* disk./.usagePercent‚19.304461373665156 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 404.00390625 disk.sda1.out‚0% proc.memory.availableMB‚ 330.96875 proc.cpu.threads‚17 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.out‚0 network.sent‚ 2.7587395e+07 disk.drbd2265.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚74146.9208984375# proc.memory.rssMB‚ 2952.33203125 cpu‚25.26837852883742 disk.drbd1716.in‚0 disk.sda3.out‚ 404.00390625 disk.sda.in‚0.203125 disk.sda.out‚ 404.00390625 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 disk.rbd1.out‚ 9.818359375 disk.drbd2080.in‚0* proc.memory.percent‚18.019604682922363$ disk./.usageGB‚287.6504249572754 disk.rbd1.in‚0" disk.dm-0.out‚74146.9208984375$ memory_percent‚97.97992706298828 disk.rbd0.in‚0% disk.nvme0n1.out‚74146.9208984375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125+ã&Ï4õ)/ _wandbruntime‚10833  _runtime‚10833ã\lKÈö) trainer/global_step‚10799 % _timestamp‚1.7884388555539308e+09 _runtime‚10833.90895308 _step‚884 & lr-AdamW/pg1‚7.023658296423097e-06 & lr-AdamW/pg2‚7.023658296423097e-06G9Í÷)Ç _step‚884 & lr-AdamW/pg1‚7.023658296423097e-06 & lr-AdamW/pg2‚7.023658296423097e-06 trainer/global_step‚10799 % _timestamp‚1.7884388555539308e+09 _runtime‚10833.90895308ô«QC|4ø)/ _wandbruntime‚10833  _runtime‚10833âÆ~å$(ù)žP aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1465332303533056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 trainer/global_step‚10799 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193328103 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23065349382656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1463369070414469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1465332303533056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23065349382656 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1252035378923008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200324255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚10833.909893455 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232381084 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115607037 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31988219128832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1440342711763968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1488397626524160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1488400758309376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232404190 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193328103 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23029462119425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23026358650501 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1488397626524160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84717218 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚885 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 200324806 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23068454776320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425732299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 425709187 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1488400758309376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1465332277141504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84717766 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23065349382656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1463369070414469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1465332303533056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23065349382656 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1252035378923008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200324255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚10833.909893455 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232381084 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115607037 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚31988219128832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1440342711763968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1488397626524160 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1488400758309376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232404190 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193328103 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23029462119425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23026358650501 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1488397626524160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84717218 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 _step‚885 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 200324806 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23068454776320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425732299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 425709187 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1488400758309376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1465332277141504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84717766 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84725439 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1252147817921536 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1463503705105885 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200342206 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232402174 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1465463797621248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425747561 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232425306 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end0ŠãÔ/segment.small_pool.freed‚0 % _timestamp‚1.7884388564480767e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1465463771229696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193345387 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1463500576208785 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23067461696512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232425306 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115617290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425770699 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚886 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1440472108765184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1284138945845760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31991070825984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425770699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232402174 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1488534364724736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23067461696512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23070567103488 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640dÔ^m3'ý)­N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23070567103488 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200342732 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115617293 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚31991127924224 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚10834.803680935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425747561 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193345393 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1252147789787136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84725439 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1252147817921536 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1463503705105885 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200342206 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232402174 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1465463797621248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425747561 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232425306 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884388564480767e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1465463771229696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193345387 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1463500576208785 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23067461696512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232425306 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115617290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425770699 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _step‚886 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1440472108765184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1284138945845760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚31991070825984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425770699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232402174 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1488534364724736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23067461696512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878öùaÕZ4þ)/ _wandbruntime‚10834  _runtime‚10834ðÖd”`ÿ)Ú $ _timestamp‚1.788438856449902e+09 0loss/FRt3/assignment_loss‚2.7814453878710346 /loss/FBt4/detection_loss‚1.5203662785616787 %loss/total_loss‚43.02113952223607 0loss/FRt4/assignment_loss‚2.5303718977504306 2loss/SRqqt1/assignment_loss‚1.0215224790821473 2loss/SRqqt3/assignment_loss‚1.0970496911444325 3loss/FBt1/assignment_loss‚0.0009334440609212872 /loss/FRt1/detection_loss‚1.6488657249664922 .loss/FBt3/detection_loss‚2.446137860417366 2loss/SRqqt4/assignment_loss‚1.4552634693682194 /loss/FRt2/detection_loss‚2.4821279488858723 _runtime‚10834.804874588 .loss/FRt3/detection_loss‚2.895428960361788 .loss/FRt1/assignment_loss‚1.59014315987533 /loss/FRt2/assignment_loss‚3.166151909601121 1loss/FBt3/assignment_loss‚0.37785656942227774 1loss/SRqqt2/detection_loss‚2.3490893244743347 1loss/SRqqt4/detection_loss‚3.0820408195257185 epoch‚8 2loss/SRqqt2/assignment_loss‚1.9397913463986838 0loss/FBt2/assignment_loss‚0.6047698009497253 1loss/FBt4/assignment_loss‚0.14997426206669348 /loss/FBt1/detection_loss‚0.5725078308628267 /loss/FBt2/detection_loss‚3.2309699803590775 _step‚887 /loss/FRt4/detection_loss‚3.6325437194771237 1loss/SRqqt1/detection_loss‚1.0350292405982813 1loss/SRqqt3/detection_loss‚1.4107584161544913 trainer/global_step‚10799ÌÑ}+e€*ß 1loss/FBt4/assignment_loss‚0.14997426206669348 /loss/FBt1/detection_loss‚0.5725078308628267 /loss/FBt2/detection_loss‚3.2309699803590775 _step‚887 /loss/FRt4/detection_loss‚3.6325437194771237 1loss/SRqqt1/detection_loss‚1.0350292405982813 1loss/SRqqt3/detection_loss‚1.4107584161544913 trainer/global_step‚10799 $ _timestamp‚1.788438856449902e+09 0loss/FRt3/assignment_loss‚2.7814453878710346 /loss/FBt4/detection_loss‚1.5203662785616787 %loss/total_loss‚43.02113952223607 0loss/FRt4/assignment_loss‚2.5303718977504306 2loss/SRqqt1/assignment_loss‚1.0215224790821473 2loss/SRqqt3/assignment_loss‚1.0970496911444325 3loss/FBt1/assignment_loss‚0.0009334440609212872 /loss/FRt1/detection_loss‚1.6488657249664922 .loss/FBt3/detection_loss‚2.446137860417366 2loss/SRqqt4/assignment_loss‚1.4552634693682194 /loss/FRt2/detection_loss‚2.4821279488858723 _runtime‚10834.804874588 .loss/FRt3/detection_loss‚2.895428960361788 .loss/FRt1/assignment_loss‚1.59014315987533 /loss/FRt2/assignment_loss‚3.166151909601121 1loss/FBt3/assignment_loss‚0.37785656942227774 1loss/SRqqt2/detection_loss‚2.3490893244743347 1loss/SRqqt4/detection_loss‚3.0820408195257185 epoch‚8 2loss/SRqqt2/assignment_loss‚1.9397913463986838 0loss/FBt2/assignment_loss‚0.6047698009497253÷7/‘*‚(:† ÒÒåԬضµ gpu.0.gpu‚83.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚263.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pPÁgw‚*‚(:ì ÒÒåÔñ𥷠disk.nvme0n1.in‚ 471.90234375) disk./.usagePercent‚19.30618276832539 disk.sda2.out‚0% proc.memory.availableMB‚ 330.90625 network.sent‚ 2.7593237e+07% disk./.usageGB‚287.67607498168945 disk.drbd2080.in‚0 disk.dm-1.out‚ 404.45703125 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0 disk.sda.in‚0.203125% disk.nvme0n1.out‚74219.6044921875 cpu‚25.268098599193994 disk.drbd2265.out‚0" disk.dm-0.out‚74219.6044921875 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.drbd2265.in‚0$ memory_percent‚97.98030853271484* proc.memory.percent‚18.019604682922363 disk.sda3.out‚ 404.45703125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2952.33203125 proc.cpu.threads‚17 network.recv‚ 3.644317e+06 disk.rbd1.in‚0' disk.nvme0c0n1.out‚74219.6044921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.out‚ 9.818359375 disk.drbd1716.in‚0 disk.sda.out‚ 404.45703125B/“ƒ*‚(:ˆ áÒåÔƒõñµ gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚278.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 2Sv„*‚(:ë áÒåÔí€ì· disk.sda1.in‚0& disk.nvme0n1.out‚74296.11474609375 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 405.015625( proc.memory.availableMB‚ 330.66796875 network.recv‚ 3.65132e+06 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 471.90234375$ memory_percent‚97.98176288604736# proc.memory.rssMB‚ 2952.33203125 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 network.sent‚ 2.7702422e+07 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 9.8505859375 cpu‚25.26779547738661 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 405.015625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚74296.11474609375 disk.sda3.in‚0.203125 disk.dm-0.in‚ 471.90234375* proc.memory.percent‚18.019604682922363 disk.drbd1716.in‚0 disk.sda.out‚ 405.015625 disk.rbd0.in‚0# disk.dm-0.out‚74296.11474609375* disk./.usagePercent‚19.306909829722628$ disk./.usageGB‚287.6869087219238 disk.drbd2265.in‚0 disk.sda2.out‚0ˆâ|ž…*‚(:“ ðÒåÔ«Ëí´ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚290.708' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.141600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ãw†*‚(:ì ðÒåÔ­ÔÏ· disk.drbd2266.in‚0( disk.nvme0c0n1.out‚74361.86572265625 disk.drbd1716.in‚0 disk.sda.in‚0.203125 network.recv‚ 3.653972e+06 disk.sda3.out‚ 405.4765625 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 471.90234375$ memory_percent‚97.98030853271484% proc.memory.availableMB‚ 330.90625# proc.memory.rssMB‚ 2952.33203125 disk.dm-1.out‚ 405.4765625 disk.dm-0.in‚ 471.90234375 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.267496856261555 network.sent‚ 2.7706782e+07) disk./.usagePercent‚19.30712359601372$ disk./.usageGB‚287.6900939941406 disk.rbd1.in‚0 disk.rbd1.out‚ 9.8505859375 disk.sda1.in‚0* proc.memory.percent‚18.019604682922363 proc.cpu.threads‚17 disk.drbd2266.out‚0# disk.dm-0.out‚74361.86572265625 disk.sda.out‚ 405.4765625 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125& disk.nvme0n1.out‚74361.86572265625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0%Á×4‡*/ _wandbruntime‚10881  _runtime‚10881®wšÞȈ* & lr-AdamW/pg2‚6.495868881737463e-06 trainer/global_step‚10849 $ _timestamp‚1.788438902864825e+09 _runtime‚10881.219939138 _step‚888 & lr-AdamW/pg1‚6.495868881737463e-06xÈÖ͉*Ç & lr-AdamW/pg2‚6.495868881737463e-06 trainer/global_step‚10849 $ _timestamp‚1.788438902864825e+09 _runtime‚10881.219939138 _step‚888 & lr-AdamW/pg1‚6.495868881737463e-06ø¥«¤d4Š*/ _wandbruntime‚10881  _runtime‚10881ƒüŸšm#‹*žP KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116119690 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚85097937 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1495077961040384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194192309 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1495081092825600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 % _timestamp‚1.7884389028656273e+09 _runtime‚10881.220921994 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 trainer/global_step‚10849 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194192309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1495077961040384 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1289790189777408 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1289790104531456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1495081092825600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1469944373721477 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 427631239 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23170979489280 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23131811896709 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201217624 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1469947502618369 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1471907007942656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233438936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 233462042 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1257655922129408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1471906981551104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1471907007942656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 194192303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1446812587252736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1446812561824768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_bB58·atch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 233438936 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 233462042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85098480 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1495081092825600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1469944373721477 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 427631239 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23170979489280 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23131811896709 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201217624 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1469947502618369 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1471907007942656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233438936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 233462042 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1257655922129408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1471906981551104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1471907007942656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 194192303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1446812587252736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1446812561824768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 233438936 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 233462042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85098480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1495077961040384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194192309 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1495081092825600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 % _timestamp‚1.7884389028656273e+09 _runtime‚10881.220921994 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 trainer/global_step‚10849 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194192309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1495077961040384 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1289790189777408 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1289790104531456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456ùƨ˜4*/ _wandbruntime‚10882  _runtime‚10882ä… !.'Ž*¨N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427669613 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32137097859584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194209587 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427692751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85105404 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1257768332993536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1472038475639296 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194209593 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1446941984253952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194209593 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚10849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233483158 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427692751 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1472038502030848 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 427669613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23133920689809 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23173091803136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85105933 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1495214699240960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1470079008412893 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1472038502030848 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23176197210112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1289905430853120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1257768361127936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201235344 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23176197210112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233483158 % _timestamp‚1.7884389037590735e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚10882.114566693 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1495211567442432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738©'¯43'*­N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1257768361127936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201235344 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23176197210112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233483158 % _timestamp‚1.7884389037590735e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _runtime‚10882.114566693 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1495211567442432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427669613 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32137097859584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194209587 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427692751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85105404 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1257768332993536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1472038475639296 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194209593 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1446941984253952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194209593 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 trainer/global_step‚10849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233483158 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427692751 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1472038502030848 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 427669613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23133920689809 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23173091803136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85105933 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1495214699240960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1470079008412893 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1472038502030848 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23176197210112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1289905430853120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ú½ ;4*/ _wandbruntime‚10882  _runtime‚10882ô²[¤&‘*× 1loss/SRqqt1/detection_loss‚0.9120303285446497 0loss/SRqqt2/detection_loss‚2.215857499279082 .loss/FBt3/detection_loss‚3.059161206086477 .loss/FRt2/detection_loss‚2.652547071661268 .loss/FRt3/detection_loss‚2.537193772279554 .loss/FRt4/detection_loss‚3.148445834284243 epoch‚8 0loss/FRt1/assignment_loss‚1.1949001230410676 2loss/SRqqt3/assignment_loss‚1.5669774029660841 %loss/total_loss‚41.53636486343048 /loss/FRt1/detection_loss‚1.6700877528359641 % _timestamp‚1.7884389037607996e+09 1loss/FBt2/assignment_loss‚0.07246573318261654 /loss/FBt4/detection_loss‚2.2815856704345117 trainer/global_step‚10849 1loss/FBt1/assignment_loss‚0.03208481125682994 2loss/FBt3/assignment_loss‚0.021657751163094947 0loss/FRt3/assignment_loss‚2.2070692860417895 0loss/FRt4/assignment_loss‚3.2865122712176777 1loss/SRqqt2/assignment_loss‚1.757770802360028 /loss/FBt1/detection_loss‚0.6203105949023138 /loss/FBt2/detection_loss‚1.9944389502207438 _runtime‚10882.116072836 1lossCH÷7/SRqqt3/detection_loss‚1.7482187768508648 1loss/SRqqt4/detection_loss‚2.4169071767781234 _step‚891 0loss/FBt4/assignment_loss‚0.2588649692278033 /loss/FRt2/assignment_loss‚3.373925701209477 2loss/SRqqt1/assignment_loss‚0.8894782128817595 1loss/SRqqt4/assignment_loss‚1.617873164724458;çËb’*Ü 1loss/FBt2/assignment_loss‚0.07246573318261654 /loss/FBt4/detection_loss‚2.2815856704345117 trainer/global_step‚10849 1loss/FBt1/assignment_loss‚0.03208481125682994 2loss/FBt3/assignment_loss‚0.021657751163094947 0loss/FRt3/assignment_loss‚2.2070692860417895 0loss/FRt4/assignment_loss‚3.2865122712176777 1loss/SRqqt2/assignment_loss‚1.757770802360028 /loss/FBt1/detection_loss‚0.6203105949023138 /loss/FBt2/detection_loss‚1.9944389502207438 _runtime‚10882.116072836 1loss/SRqqt3/detection_loss‚1.7482187768508648 1loss/SRqqt4/detection_loss‚2.4169071767781234 _step‚891 0loss/FBt4/assignment_loss‚0.2588649692278033 /loss/FRt2/assignment_loss‚3.373925701209477 2loss/SRqqt1/assignment_loss‚0.8894782128817595 1loss/SRqqt4/assignment_loss‚1.617873164724458 1loss/SRqqt1/detection_loss‚0.9120303285446497 0loss/SRqqt2/detection_loss‚2.215857499279082 .loss/FBt3/detection_loss‚3.059161206086477 .loss/FRt2/detection_loss‚2.652547071661268 .loss/FRt3/detection_loss‚2.537193772279554 .loss/FRt4/detection_loss‚3.148445834284243 epoch‚8 0loss/FRt1/assignment_loss‚1.1949001230410676 2loss/SRqqt3/assignment_loss‚1.5669774029660841 %loss/total_loss‚41.53636486343048 /loss/FRt1/detection_loss‚1.6700877528359641 % _timestamp‚1.7884389037607996e+09û§«­œ“*‚(:‘ ÿÒåÔ£“µµ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚300.154' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.03079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°IàWy”*‚(:î ÿÒåÔÇŠ´· disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 406.328125( proc.memory.availableMB‚ 331.49609375 cpu‚25.267176072802695 disk.sda2.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2952.33203125 network.recv‚ 3.658857e+06* disk./.usagePercent‚19.308875199562994 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0* proc.memory.percent‚18.019604682922363 network.sent‚ 2.771295e+07 disk.sda3.in‚0.203125% disk./.usageGB‚287.71619415283203 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.out‚ 406.328125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 406.328125 disk.sda.in‚0.203125" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚74455.30224609375 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚74455.30224609375 disk.drbd2265.in‚0$ memory_percent‚97.97670841217041& disk.nvme0n1.out‚74455.30224609375 disk.rbd1.out‚ 10.2724609375 disk.sda1.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0Øj^i“•*‚(:ˆ ŽÓåÔ‡‚„µ gpu.0.gpu‚87.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚305.996' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.1992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®?"w–*‚(:ì ŽÓåÔÓಶ* proc.memory.percent‚18.019604682922363 cpu‚25.266882926548146 disk.dm-0.in‚ 471.90234375 disk.drbd2265.in‚0& disk.nvme0n1.out‚74514.07568359375 disk.sda2.in‚0 disk.sda3.in‚0.203125 disk.sda.out‚ 406.69140625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 network.sent‚ 2.7821825e+07$ disk./.usageGB‚287.6997756958008 disk.drbd2080.in‚0 disk.sda.in‚0.203125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 471.90234375# proc.memory.rssMB‚ 2952.33203125" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2266.out‚0 disk.rbd1.out‚ 10.2724609375* disk./.usagePercent‚19.307773343135622 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 406.69140625 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚74514.07568359375 disk.sda3.out‚ 406.69140625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0# memory_percent‚97.9766845703125! proc.memory.availableMB‚331.5 network.recv‚ 3.665393e+06# disk.dm-0.out‚74514.07568359375 disk.sda1.out‚0 disk.drbd2265.out‚0imlœ—*‚(:‘ ÓåÔâíè´ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.335' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.866999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 bMƒf˜*‚(:Û ÓåÔÉ‹Ö· proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.out‚0" disk.dm-0.out‚74570.2744140625 disk.sda3.in‚0.203125 disk.sda3.out‚407.375 disk.drbd2265.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚407.375% disk.nvme0n1.out‚74570.2744140625 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.drbd1716.in‚0 network.recv‚ 3.669885e+06 disk.sda2.out‚0 disk.dm-1.out‚407.375 disk.rbd1.in‚0 disk.drbd2080.in‚0 cpu‚25.27007413565627 disk.rbd0.out‚0 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚74570.2744140625 network.sent‚ 2.782834e+07 disk.drbd2265.out‚0" disk.nvme0c0n1.in‚ 471.90234375) disk./.usagePercent‚19.30978735440749$ memory_percent‚97.97942638397217 proc.memory.rssMB‚2952.375* proc.memory.percent‚18.019866943359375% disk./.usageGB‚287.72978591918945 disk.rbd1.out‚ 10.4287109375 disk.sda1.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 331.05078125âúaq›™*‚(: ¬ÓåÔ¢òÒµ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚297.443' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.48859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ`¥qš*‚(:æ ¬ÓåÔž·ô· network.recv‚ 3.673224e+06* disk./.usagePercent‚19.310591730080073 disk.drbd1716.in‚0 disk.sda.out‚ 407.828125 disk.drbd2265.out‚0 disk.sda2.in‚0% disk./.usageGB‚287.74177169799805 disk.dm-0.in‚ 471.90234375( proc.memory.availableMB‚ 439.39453125* proc.memory.percent‚18.019866943359375 network.sent‚ 2.7833663e+07 disk.rbd1.in‚0" disk.dm-0.out‚74649.1376953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚74649.1376953125 disk.sda1.in‚0% disk.nvme0n1.out‚74649.1376953125 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.out‚ 10.4287109375" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-1.out‚ 407.828125 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda3.out‚ 407.828125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.31814861297607 disk.drbd1716.out‚0 disk.sda2.out‚0 proc.memory.rssMB‚2952.375 cpu‚25.27542836208128 proc.cpu.threads‚17 disk.sda1.out‚0nup‘›*‚(:† »ÓåÔ´ê´ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚40.0 gpu.0.powerWatts‚123.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãf­ä}œ*‚(:ò »ÓåÔˆ¹Ó·* disk./.usagePercent‚19.311571214906078 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 10.5849609375" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.dm-1.out‚ 408.51171875( disk.nvme0c0n1.out‚74722.48681640625 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0.203125$ memory_percent‚96.87685966491699 cpu‚25.281134766012197 proc.cpu.threads‚17# disk.dm-0.out‚74722.48681640625 disk.drbd2265.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.579030990600586& disk.nvme0n1.out‚74722.48681640625 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 408.51171875 disk.sda3.out‚ 408.51171875 disk.drbd1716.out‚0' proc.memory.availableMB‚ 511.6953125" proc.memory.rssMB‚ 2880.1484375$ disk./.usageGB‚287.7563667297363 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.7840302e+07 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 network.recv‚ 3.678579e+06b‰Žº‘*‚(:† ÊÓåÔ¨š÷´ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¶þzž*‚(:ï ÊÓåÔ¬ÌÙ· disk.drbd2266.in‚0 proc.memory.rssMB‚2952.375 proc.cpu.threads‚17% disk./.usageGB‚287.78027725219727 disk.sda1.out‚0 cpu‚25.286414716081286 network.sent‚ 2.7846132e+07# disk.dm-0.out‚74761.29541015625 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2265.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚74761.29541015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.5849609375 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 409.08203125$ memory_percent‚97.31955528259277' proc.memory.availableMB‚ 439.1640625 disk.dm-0.in‚ 471.90234375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 409.08203125 disk.dm-1.in‚0.203125 disk.sda3.out‚ 409.08203125 disk.nvme0n1.in‚ 471.90234375 network.recv‚ 3.682731e+06* disk./.usagePercent‚19.313175870130696 disk.sda.in‚0.203125* proc.memory.percent‚18.019866943359375( disk.nvme0c0n1.out‚74761.29541015625 disk.drbd2265.out‚0Yt4Ÿ*/ _wandbruntime‚10966  _runtime‚10966QËâ* *‡U J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 111478272 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚1478342566570269 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1503768605466112 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1480504832350208 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1503770959367168 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 % _timestamp‚1.7884389877073872e+09 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚85603255 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1503770959367168 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 trainer/global_step‚10880 cNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚1480504856644608 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚23266102722560 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1503768605466112 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚23266102722560 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 429888485 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚85603119 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1700 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚23224417175237 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚971 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 129126912 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 fQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚1455115821704192 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 429906768 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 429906768 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 429888485 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚136 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1264941639804416 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚1478340215548613 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚32252456567296 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 116921322 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 234496391 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚138 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 234514669 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1297194096371712 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 _step‚892 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 195392099 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 195392094 _runtime‚10966.062948164 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 234496391 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚23263773115904 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1455115798373376 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878J1é)¡*ŒU bMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚1480504832350208 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 202524439 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚23226744866077 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 234514669 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1878 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1302 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18278 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2353901056 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1297194225498624 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 195392099 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202524577 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚738 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 111478272 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚1478342566570269 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1503768605466112 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1480504832350208 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1503770959367168 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 % _timestamp‚1.7884389877073872e+09 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3329 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚85603255 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1503770959367168 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1302 trainer/global_step‚10880 cNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚1480504856644608 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚23266102722560 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1503768605466112 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚23266102722560 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2329606656 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 429888485 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚85603119 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1700 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚23224417175237 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚971 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 129126912 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 fQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚1455115821704192 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 429906768 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 429906768 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2353901056 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 429888485 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚136 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1264941639804416 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚1478340215548613 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚32252456567296 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3180 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3329 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 116921322 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 234496391 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚138 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 234514669 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1297194096371712 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 _step‚892 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 195392099 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 195392094 _runtime‚10966.062948164 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 234496391 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚23263773115904 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1455115798373376 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1503858696093184 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1503858696093184 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1480594449254400 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021656 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 195405297 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚23264271133184 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚137 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 28510781440 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 dODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚1455200771304448 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚23266600739840 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 234522363 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1480594424960000 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 234504085 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1265016916529664 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 195405292 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1297270000241152 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1297269871114240 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 429909377 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111478272 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚23224913626885 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 195405297 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18278 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788438988791779e+09 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1480594449254400 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 28510781440 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 195405292 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878632448 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 234522363 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚139 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 234504085 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 202539538 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1480594424960000 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚85608415 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚32252954584576 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1478428012622173 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 129126912 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 202539538 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3180 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1480594424960000 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚85608415 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚32252954584576 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1478428012622173 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 129126912 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 21432077312 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3180 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1503858696093184ý Û¸’w¥*‚(:ì ÙÓåÔ½ÍÓ· disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 439.4140625 proc.cpu.threads‚17& disk.nvme0n1.out‚74833.34619140625 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 409.58203125 disk.drbd2266.out‚0% disk./.usageGB‚287.79846572875977# disk.dm-0.out‚74833.34619140625 disk.sda3.out‚ 409.58203125 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 proc.memory.rssMB‚2952.375 network.recv‚ 3.687752e+06* disk./.usagePercent‚19.314396514053954 disk.dm-0.in‚ 471.90234375 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.in‚0 disk.rbd1.in‚0$ memory_percent‚97.31802940368652 cpu‚25.292362156437388 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.019866943359375 network.sent‚ 2.7852447e+07 disk.sda1.in‚0 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚74833.34619140625 disk.drbd2080.in‚0 disk.sda.out‚ 409.58203125 disk.rbd1.out‚ 10.6171875ÿN+ ž¦*‚(:“ ÙÓåÔÓýË· gpu.0.gpu‚98.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚41.0 gpu.0.powerWatts‚139.189' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.837799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00š˜f4§*/ _wandbruntime‚10981  _runtime‚10981I¹“Vq%¨*Ñ¥ 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4991950988769531 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5457502007484436 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0029846697580069304 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 'particle/f_score‚0.3713664412498474 particle/accuracy_6_of_7‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 jet/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22119343280792236 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4992011785507202 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2549729347229004 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2204202562570572 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5557311773300171 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.40740740299224854 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702128499746323 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02467358112335205 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27310851216316223 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013725490309298038 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15793651342391968 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3510015308856964 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5491717457771301 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10651833564043045 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_7_of_7‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6956697106361389 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4118943512439728 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4014548063278198 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2546296715736389 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4895833134651184 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48901885747909546 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1764896661043167 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21924708783626556 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011375661939382553 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 epoch‚8 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.6410256028175354 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1423003226518631 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4065536856651306 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚894 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5738070607185364 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21381613612174988 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4225574731826782 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0032712656538933516 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39610257744789124 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48627927899360657 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.991405189037323 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39836832880973816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3955135941505432 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00802671816200018 +jet/accuracy_2_of_2‚0.10220060497522354 particle/accuracy_4_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4006944000720978 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8131454586982727 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3505703806877136 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5396551489830017 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19662845134735107 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23017241060733795 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_3_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3444444239139557 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9933088421821594 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9933648109436035 7¡eÑ£gm Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_5_of_5‚1 trainer/global_step‚10880 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3818168044090271 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9881848096847534 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3884187340736389 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2420705407857895 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.40728920698165894 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20102931559085846 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.35185182094573975 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26864203810691833 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4343915283679962 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08241041749715805 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2714538872241974 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.604977011680603 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.29242900013923645 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2129545509815216 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015723271295428276 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04058543220162392 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43591657280921936 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 particle/accuracy_5_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185670852661 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17917513847351074 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.28029319643974304 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5562499165534973 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992542564868927 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7954545617103577 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5471735000610352 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5755953192710876 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23652097582817078 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23139870166778564 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4662289321422577 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35885417461395264 particle/accuracy_1_of_3‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04532267525792122 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25746554136276245 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.29798850417137146 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 )loss/val_total_loss‚41.29020945261539 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29771745204925537 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.558195948600769 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003652344224974513 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3162309229373932 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022450698539614677 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3337690234184265 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23989897966384888 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5453246235847473 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9478346109390259 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.017864791676402092 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2968498468399048 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03044217638671398 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933352470397949 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2804111838340759 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012384391389787197 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21759259700775146 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5608505606651306 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09967920184135437 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.47240787744522095 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3509184718132019 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4277116656303406 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941194653511047 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28636234998703003 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 *jet/accuracy_1_of_4‚0.8964470028877258 *jet/accuracy_2_of_4‚0.3552309572696686 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022437723353505135 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 % _timestamp‚1.7884390031974947e+09 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.35753628611564636 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25146445631980896 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9461981654167175 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3421981930732727 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09943366795778275 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5466269254684448 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.019265232607722282 particle/accuracy_3_of_5‚1 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.36936938762664795 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14035087823867798 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15567058324813843 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.40625447034835815 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4301808178424835 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20532363653182983 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.34685105085372925 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25828590989112854 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4080459773540497 particle/accuracy_2_of_4‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25929075479507446 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34853801131248474 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3598078191280365 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43023255467414856 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23119603097438812 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.21966120600700378 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5760088562965393 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4125449061393738 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922927618026733 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 jet/accuracy_5_of_7‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581240415573 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3955620229244232 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 7"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.45333331823349 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990437388420105 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7803030610084534 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_3_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4720763564109802 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.37075644731521606 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.24425287544727325 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34876543283462524 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4072326123714447 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3317815363407135 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26205745339393616 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8454912304878235 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.34229087829589844 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.39806246757507324 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3547881841659546 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7081426382064819 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424240708351135 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.21909703314304352 ,jet/accuracy_4_of_4‚0.001779701909981668 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.544751763343811 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18572372198104858 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.40978673100471497 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995679259300232 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08392857015132904 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017233725637197495 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.368486613035202 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23986487090587616 +jet/accuracy_3_of_4‚0.07234679907560349 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_2_of_7‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.243149071931839 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8417818546295166 +jet/accuracy_3_of_5‚0.09553250670433044 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25723785161972046 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3747684955596924 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2865496873855591 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3619888126850128 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39632150530815125 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19111110270023346 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934348464012146 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41124555468559265 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44188594818115234 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.36890101432800293 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06325426697731018 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.569288969039917 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6397919654846191 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3371386229991913 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15988793969154358 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18531657755374908 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.43081575632095337 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3505703806877136 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 _runtime‚10981.555061741 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25547099113464355 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2755809724330902 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.518822968006134 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36175599694252014 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.033694423735141754 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333332419395447 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18662294745445251 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1851884126663208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_4_of_6‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 jet/accuracy_3_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1851884126663208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24035686254501343 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23620787262916565 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2208620309829712 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02007366716861725 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19826295971870422 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37924104928970337 particle/accuracy_2_of_5‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16355377435684204 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9875672459602356 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375ºQ+Æ‹©*Ö¥ A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3747684955596924 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2865496873855591 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3619888126850128 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39632150530815125 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19111110270023346 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934348464012146 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41124555468559265 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44188594818115234 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.36890101432800293 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06325426697731018 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.569288969039917 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6397919654846191 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3371386229991913 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15988793969154358 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18531657755374908 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.43081575632095337 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3505703806877136 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 _runtime‚10981.555061741 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25547099113464355 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2755809724330902 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.518822968006134 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36175599694252014 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.033694423735141754 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333332419395447 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18662294745445251 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1851884126663208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_4_of_6‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 jet/accuracy_3_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1851884126663208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.240356862545Ïeî«ù01343 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23620787262916565 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2208620309829712 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02007366716861725 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19826295971870422 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37924104928970337 particle/accuracy_2_of_5‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16355377435684204 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9875672459602356 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4991950988769531 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5457502007484436 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0029846697580069304 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 'particle/f_score‚0.3713664412498474 particle/accuracy_6_of_7‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 jet/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22119343280792236 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4992011785507202 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2549729347229004 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2204202562570572 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5557311773300171 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.40740740299224854 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702128499746323 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02467358112335205 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27310851216316223 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013725490309298038 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15793651342391968 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3510015308856964 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5491717457771301 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10651833564043045 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_7_of_7‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6956697106361389 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4118943512439728 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4014548063278198 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2546296715736389 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4895833134651184 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48901885747909546 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1764896661043167 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21924708783626556 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011375661939382553 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 epoch‚8 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.6410256028175354 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1423003226518631 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4065536856651306 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚894 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5738070607185364 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21381613612174988 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4225574731826782 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0032712656538933516 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39610257744789124 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48627927899360657 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.991405189037323 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39836832880973816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3955135941505432 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00802671816200018 +jet/accuracy_2_of_2‚0.10220060497522354 particle/accuracy_4_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4006944000720978 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8131454586982727 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3505703806877136 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5396551489830017 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19662845134735107 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23017241060733795 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_3_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3444444239139557 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9933088421821594 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9933648109436035 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_5_of_5‚1 trainer/global_step‚10880 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3818168044090271 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9881848096847534 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3884187340736389 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2420705407857895 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.40728920698165894 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20102931559085846 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.35185182094573975 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26864203810691833 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4343915283679962 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08241041749715805 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2714538872241974 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.604977011680603 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.29242900013923645 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2129545509815216 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015723271295428276 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04058543220162392 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43591657280921936 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 particle/accuracy_5_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185670852661 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17917513847351074 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.28029319643974304 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5562499165534973 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992542564868927 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7954545617103577 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5471735000610352 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5755953192710876 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23652097582817078 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23139870166778564 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4662289321422577 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35885417461395264 particle/accuracy_1_of_3‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04532267525792122 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25746554136276245 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.29798850417137146 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 )loss/val_total_loss‚41.29020945261539 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29771745204925537 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.558195948600769 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003652344224974513 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3162309229373932 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022450698539614677 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3337690234184265 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23989897966384888 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5453246235847473 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9478346109390259 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.017864791676402092 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2968498468399048 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03044217638671398 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933352470397949 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2804111838340759 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012384391389787197 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21759259700775146 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5608505606651306 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09967920184135437 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.47240787744522095 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3509184718132019 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4277116656303406 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941194653511047 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28636234998703003 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 *jet/accuracy_1_of_4‚0.8964470028877258 *jet/accuracy_2_of_4‚0.3552309572696686 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022437723353505135 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 % _timestamp‚1.7884390031974947e+09 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.35753628611564636 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25146445631980896 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9461981654167175 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3421981930732727 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09943366795778275 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5466269254684448 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.019265232607722282 particle/accuracy_3_of_5‚1 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.36936938762664795 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14035087823867798 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15567058324813843 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.40625447034835815 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4301808178424835 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20532363653182983 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.34685105085372925 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25828590989112854 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4080459773540497 particle/accuracy_2_of_4‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25929075479507446 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34853801131248474 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3598078191280365 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43023255467414856 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23119603097438812 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.21966120600700378 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5760088562965393 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4125449061393738 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922927618026733 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 jet/accuracy_5_of_7‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581240415573 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3955620229244232 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 7"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.45333331823349 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990437388420105 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7803030610084534 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_3_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4720763564109802 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.37075644731521606 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.24425287544727325 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34876543283462524 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4072326123714447 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3317815363407135 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26205745339393616 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8454912304878235 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.34229087829589844 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.39806246757507324 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3547881841659546 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7081426382064819 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424240708351135 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.21909703314304352 ,jet/accuracy_4_of_4‚0.001779701909981668 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.544751763343811 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18572372198104858 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.40978673100471497 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995679259300232 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08392857015132904 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017233725637197495 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.368486613035202 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23986487090587616 +jet/accuracy_3_of_4‚0.07234679907560349 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_2_of_7‚0.5 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.243149071931839 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8417818546295166 +jet/accuracy_3_of_5‚0.09553250670433044 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25723785161972046 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0K–ÞY ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0þhÄzÉ‘ª*‚(:† èÓåÔڳõ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.019' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2038 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§¤¹v«*‚(:ë èÓåÔŸËâ· cpu‚25.291328507374523& disk.nvme0n1.out‚80308.99951171875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0* disk./.usagePercent‚19.438579624733094$ disk./.usageGB‚289.6488838195801 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.dm-1.out‚ 410.078125 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.922974586486816 network.sent‚ 2.7960235e+07 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.out‚0 disk.rbd1.out‚ 10.6171875 disk.sda3.out‚ 410.078125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0$ memory_percent‚72.71018028259277 proc.cpu.threads‚17 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚80308.99951171875# disk.dm-0.out‚80308.99951171875 disk.drbd1716.in‚0 disk.sda.out‚ 410.078125( proc.memory.availableMB‚ 4471.1640625 network.recv‚ 3.693055e+06 disk.drbd2266.in‚0 disk.dm-0.in‚ 471.90234375# proc.memory.rssMB‚ 2772.66015625Á‹$s“¬*‚(:ˆ ÷ÓåÔ’ã…µ gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚291.244' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/æC'{­*‚(:ð ÷ÓåÔçÞ¶ disk.sda1.out‚0 disk.rbd1.out‚ 10.7734375" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.out‚ 410.73828125* proc.memory.percent‚17.023038864135742 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-0.in‚ 471.90234375 disk.drbd2080.in‚0 disk.sda.out‚ 410.73828125 disk.drbd2265.in‚0 disk.drbd2266.in‚0$ memory_percent‚72.81122207641602 disk.sda1.in‚0( disk.nvme0c0n1.out‚81433.90087890625 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚ 3.711269e+06* disk./.usagePercent‚19.439942864852927# disk.dm-0.out‚81433.90087890625 disk.sda.in‚0.203125& disk.nvme0n1.out‚81433.90087890625 disk.dm-1.out‚ 410.73828125 disk.rbd0.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 4454.609375 network.sent‚ 2.8090458e+07 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 cpu‚25.291005382507805% disk./.usageGB‚289.66919708251953 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2789.0546875‘PØ{4®*/ _wandbruntime‚11012  _runtime‚11012yÉÍȯ* % _timestamp‚1.7884390337661226e+09 _runtime‚11012.12118645 _step‚895 & lr-AdamW/pg1‚5.988041822856397e-06 & lr-AdamW/pg2‚5.988041822856397e-06 trainer/global_step‚10899´àSÍ°*Ç _runtime‚11012.12118645 _step‚895 & lr-AdamW/pg1‚5.988041822856397e-06 & lr-AdamW/pg2‚5.988041822856397e-06 trainer/global_step‚10899 % _timestamp‚1.7884390337661226e+09ÿE’4±*/ _wandbruntime‚11012  _runtime‚11012ÄÊí€$(²*žP M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 430894536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23268558704119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195887977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1484126042667008 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117241165 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23308001675264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚32313102264832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203055471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85814791 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚85814309 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1481905924302707 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 430871424 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195887983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23311107068928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1507437149735936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234983447 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235006553 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1268018914585088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1458634236701696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195887983 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48722944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1507434017950720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235006553 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117241162 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481902795405815 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 % _timestamp‚1.7884390337670505e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195887977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234983447 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1484126016275456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23308001675264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195887983 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48722944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1507434017950720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235006553 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117241162 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481902795405815 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 % _timestamp‚1.7884390337670505e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195887977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234983447 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1484126016275456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23308001675264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23270667497219 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1507570756151296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 430932936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1482034301200131 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1458763633702912 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 430909798 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1458763659130880 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1300447395499520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117251415 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 430932936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235027669 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23313219396096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23310113989120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85821753 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203073168 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚10899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23313219396096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1507567624352768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180•Ê@Ï3'¶*­N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195905261 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1300447470246400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482037430097231 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23270667497219 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1507570756151296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 430932936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1482034301200131 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1458763633702912 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 430909798 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1458763659130880 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1300447395499520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117251415 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 430932936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235027669 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23313219396096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23310113989120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85821753 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203073168 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 trainer/global_step‚10899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23313219396096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1507567624352768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195905267 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195905261 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235027669 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46612480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203073643,’«4·*/ _wandbruntime‚11013  _runtime‚11013ßa9]¸*× /loss/FRt3/detection_loss‚2.3014710272957637 _runtime‚11013.042945924 0loss/FRt1/assignment_loss‚1.3514103456719282 0loss/FRt3/assignment_loss‚2.2650013347951377 % _timestamp‚1.7884390346880214e+09 .loss/FRt4/detection_loss‚2.780227690935135 1loss/SRqqt1/detection_loss‚0.8528897386400941 _step‚898 .loss/FBt1/detection_loss‚0.477820668856685 epoch‚9 2loss/SRqqt2/assignment_loss‚1.4823319806676845 1loss/SRqqt4/assignment_loss‚1.761204966988701 1loss/FBt1/assignment_loss‚0.07390301559302909 0loss/SRqqt3/detection_loss‚1.564848123916558 0loss/SRqqt4/detection_loss‚2.336101980163501 .loss/FBt3/detection_loss‚2.583366278145048 /loss/FRt2/detection_loss‚2.3385284005141838 2loss/SRqqt1/assignment_loss‚0.9163686658969109 1loss/FBt2/assignment_loss‚0.12327520746427277 1loss/FBt4/assignment_loss‚0.13494795014678226 1loss/SRqqt2/detection_loss‚2.6390192379122195 /loss/FBt2/detection_loss‚2.3410030245780944 trainer/global_step‚10899 0loss/FRt2/assignment_loss‚3.6286410674816225 /loss/FRt4/assignment_loss‚3.134531245512121 1loss/SRqqt3/assignment_loss‚1.166033105824941 /loss/FBt4/detection_loss‚1.8093243699807386 %loss/total_loss‚39.51165896451054 1loss/FBt3/assignment_loss‚0.03156883736657948 /loss/FRt1/detection_loss‚1.4178407001628044v+Fb¹*Ü 0loss/SRqqt3/detection_loss‚1.564848123916558 0loss/SRqqt4/detection_loss‚2.336101980163501 .loss/FBt3/detection_loss‚2.583366278145048 /loss/FRt2/detection_loss‚2.3385284005141838 2loss/SRqqt1/assignment_loss‚0.9163686658969109 1loss/FBt2/assignment_loss‚0.12327520746427277 1loss/FBt4/assignment_loss‚0.13494795014678226 1loss/SRqqt2/detection_loss‚2.6390192379122195 /loss/FBt2/detection_loss‚2.3410030245780944 trainer/global_step‚10899 0loss/FRt2/assignment_loss‚3.6286410674816225 /loss/FRt4/assignment_loss‚3.134531245512121 1loss/SRqqt3/assignment_loss‚1.166033105824941 /loss/FBt4/detection_loss‚1.8093243699807386 %loss/total_loss‚39.51165896451054 1loss/FBt3/assignment_loss‚0.03156883736657948 /loss/FRt1/detection_loss‚1.4178407001628044 /loss/FRt3/detection_loss‚2.3014710272957637 _runtime‚11013.042945924 0loss/FRt1/assignment_loss‚1.3514103456719282 0loss/FRt3/assignment_loss‚2.2650013347951377 % _timestamp‚1.7884390346880214e+09 .loss/FRt4/detection_loss‚2.780227690935135 1loss/SRqqt1/detection_loss‚0.8528897386400941 _step‚898 .loss/FBt1/detection_loss‚0.477820668856685 epoch‚9 2loss/SRqqt2/assignment_loss‚1.4823319806676845 1loss/SRqqt4/assignment_loss‚1.761204966988701 1loss/FBt1/assignment_loss‚0.07390301559302909‚±Mxžº*‚(:“ †ÔåԪ gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚295.847' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.169399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å£×s»*‚(:è †ÔåÔâ‘Ÿ disk.drbd1716.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚81491.7607421875 network.sent‚ 2.8100378e+07) disk./.usagePercent‚19.44163609868439 disk.nvme0n1.in‚ 471.90234375" proc.memory.rssMB‚ 2804.9921875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.7734375 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda.out‚ 411.12890625 disk.drbd2265.out‚0 disk.dm-1.out‚ 411.12890625 disk.sda3.out‚ 411.12890625 disk.sda1.in‚0% proc.memory.availableMB‚ 4443.8125) proc.memory.percent‚17.12031364440918 disk.sda1.out‚0 cpu‚25.290646995509654 network.recv‚ 3.722366e+06 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚81491.7607421875 disk.sda2.out‚0$ memory_percent‚72.87712097167969 disk.dm-0.in‚ 471.90234375" disk.dm-0.out‚81491.7607421875 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375$ disk./.usageGB‚289.6944274902344N,w“¼*‚(:ˆ •ÔåÔ¤‡ƒµ gpu.0.gpu‚81.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚278.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V3¢c½*‚(:Ø •ÔåÔ²Þ½¶ disk.drbd2265.in‚0 disk.rbd1.out‚ 10.8681640625" disk.nvme0c0n1.in‚ 471.90234375 network.sent‚ 2.8210604e+07 disk.sda1.in‚0$ disk./.usageGB‚289.6756706237793 disk.sda.out‚ 411.65625 disk.dm-1.out‚ 411.65625" disk.nvme0c0n1.out‚ 81557.90625 disk.sda3.out‚ 411.65625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 81557.90625 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.drbd2080.out‚0$ memory_percent‚72.94840812683105 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2821.55078125* proc.memory.percent‚17.221379280090332 disk.nvme0n1.out‚ 81557.90625 disk.dm-0.in‚ 471.90234375 disk.drbd1716.in‚0) disk./.usagePercent‚19.44037730963853 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.730719e+06 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.sda1.out‚0( proc.memory.availableMB‚ 4432.1328125 cpu‚25.290308040311004#õ¾Z‘¾*‚(:† ¤ÔåÔä£ø´ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v¾à.q¿*‚(:æ ¤ÔåÔí‘ž¹ disk.drbd2266.out‚0 disk.rbd1.out‚ 10.8681640625" disk.nvme0c0n1.in‚ 471.90234375* disk./.usagePercent‚19.441424380453576 disk.dm-1.out‚ 412.22265625$ disk.nvme0c0n1.out‚ 81628.0703125 disk.dm-0.out‚ 81628.0703125$ memory_percent‚73.02737236022949 disk.dm-0.in‚ 471.90234375 disk.sda3.out‚ 412.22265625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 4419.1953125 cpu‚25.28999219860826 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 471.90234375" disk.nvme0n1.out‚ 81628.0703125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2835.22265625) proc.memory.percent‚17.30482578277588 network.sent‚ 2.8215764e+07$ disk./.usageGB‚289.6912727355957 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 network.recv‚ 3.733872e+06 disk.sda.out‚ 412.22265625 disk.drbd2265.out‚0 disk.drbd1716.out‚0æðDÀ4À*/ _wandbruntime‚11059  _runtime‚11059¸åŸkÉÁ*à & lr-AdamW/pg1‚5.500282950177822e-06 & lr-AdamW/pg2‚5.500282950177822e-06 trainer/global_step‚10949 % _timestamp‚1.7884390810928838e+09 _runtime‚11059.447910574 _step‚899dÎÂ*È trainer/global_step‚10949 % _timestamp‚1.7884390810928838e+09 _runtime‚11059.447910574 _step‚899 & lr-AdamW/pg1‚5.500282950177822e-06 & lr-AdamW/pg2‚5.500282950177822e-06ƒâvE4Ã*/ _wandbruntime‚11059  _runtime‚11059/vlþ#(Ä*P MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 trainer/global_step‚10949 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1306100891268608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76857344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32461405342720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 _step‚900 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236064405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514114352466944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 432816588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1273639457791488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514117484252160 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1490700720685056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514114352466944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236041299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1488481227609715 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203952522 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23377115419251 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440Š+«7à Å*¢P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1306100814411264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117753812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86198707 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196752183 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514114352466944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236041299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1488481227609715 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203952522 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23377115419251 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 trainer/global_step‚10949 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1306100891268608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76857344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32461405342720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_sta£›ïert/segment.small_pool.current‚1878 _step‚900 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236064405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514114352466944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 432816588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1273639457791488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514117484252160 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1490700720685056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203969999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86205934 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1488612733404239 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1490832241164800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514247958868992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32464319519744 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1465233483763712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196769467 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1488609604507139 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1490832214773248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196769461 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236062389 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884390819878151e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 432854988 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1306216188175360 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1273751868655616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23418849502720 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196769461 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23418849502720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432854988 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚11060.343421148 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚32464368229376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 432831850 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86206381 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48709632 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 432831850 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1273751896790016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚10949 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203969999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86205934 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1488612733404239 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1490832241164800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514247958868992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32464319519744 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1465233483763712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196769467 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1488609604507139 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1490832214773248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196769461 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236062389 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 % _timestamp‚1.7884390819878151e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 432854988 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1306216188175360 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1273751868655616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23418849502720 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196769461 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23418849502720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432854988 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚11060.343421148 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚32464368229376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 432831850 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86206381 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48709632 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 432831850 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1273751896790016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚10949 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237099151 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884391285923955e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23482568665459 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237122257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1520794686983168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚32608887953408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚495 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86580527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1497275451486208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118266465 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23522367282176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204847484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1497275451486208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1471573936823296 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1520797818768384 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1311868888951296 _runtime‚11106.947941688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495053402019831 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23522367282176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197616383 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237099151 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1520797818768384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10999 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1279260000997888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32608938773504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23479465196535 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197616377 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1311868967905792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495056530916723 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1497275425094656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1497275425094656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096Ζþ)(×*£P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197616377 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1311868967905792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495056530916723 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1497275425094656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1497275425094656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚492 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237099151 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884391285923955e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23482568665459 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237122257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1520794686983168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚32608887953408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚495 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86580527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1497275451486208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118266465 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23522367282176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204847484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1497275451486208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1471573936823296 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1520797818768384 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1311868888951296 _runtime‚11106.947941688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495053402019831 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23522367282176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197616383 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237099151 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1520797818768384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10999 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1279260000997888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32608938773504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23479465196535ˆÐʪ4Ø*/ _wandbruntime‚11107  _runtime‚11107CÕÊË#Ù*¨N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237143373 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204865446 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23521374202368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1279372411862016 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1497406919182848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197633667 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1497406945574400 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1311984319833600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237120241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1471703333824512 _step‚905 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204864946 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197633667 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚500 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1311984240892416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23481573989635 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚10999 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86588728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1495188036711247 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1520931425183744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434777040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23524479609344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1520931425183744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1279372439996416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1497406945574400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1495184907814147 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434777040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118276715 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237143373 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23524479609344 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23521374202368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118276718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1471703359252480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197633661 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚11107.876515448 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884391295207076e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23484677458767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237120241 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434753902­ÈVd3'Ú*­N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23524479609344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1520931425183744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1279372439996416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50806784 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1497406945574400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1495184907814147 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434777040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118276715 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237143373 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23524479609344 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23521374202368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118276718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1471703359252480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197633661 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 _runtime‚11107.876515448 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884391295207076e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23484677458767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237120241 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434753902 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237143373 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204865446 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23521374202368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1279372411862016 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1497406919182848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197633667 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1497406945574400 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1311984319833600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237120241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1471703333824512 _step‚905 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204864946 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197633667 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚500 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1311984240892416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23481573989635 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚10999 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86588728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1495188036711247 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1520931425183744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434777040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528‰åÂø/4Û*/ _wandbruntime‚11107  _runtime‚11107õ#I:cÜ*Ý /loss/FBt2/assignment_loss‚0.553244425507728 /loss/FRt3/detection_loss‚2.3630452348126307 /loss/FBt4/detection_loss‚2.2749447524547577 _runtime‚11107.878816882 0loss/FRt2/assignment_loss‚3.0862989702890085 1loss/SRqqt3/detection_loss‚1.6832430986863263 1loss/SRqqt4/detection_loss‚2.6073945294255796 /loss/FBt2/detection_loss‚2.0797194719314573 epoch‚9 2loss/FBt1/assignment_loss‚0.017467708877567212 % _timestamp‚1.7884391295235665e+09 0loss/FRt1/assignment_loss‚1.2263612659251104 0loss/FRt3/assignment_loss‚3.2962008251084223 2loss/FBt3/assignment_loss‚0.002217652130610759 1loss/SRqqt1/detection_loss‚0.8470469637796646 trainer/global_step‚10999 _step‚906 /loss/FRt4/assignment_loss‚3.745804215299672 2loss/SRqqt3/assignment_loss‚1.2299031074035842 /loss/FRt1/detection_loss‚1.4048868778360184 1loss/SRqqt2/detection_loss‚2.2806296262048904 /loss/FBt3/detection_loss‚3.3534625470638275 2loss/SRqqt4/assignment_loss‚1.5541574967944103 .loss/FRt2/detection_loss‚2.343462366004323 /loss/FBt1/detection_loss‚0.6878256877340997 2loss/SRqqt1/assignment_loss‚0.7842284798843604 1loss/SRqqt2/assignment_loss‚1.285939568894044 2loss/FBt4/assignment_loss‚0.037327878869421384 .loss/FRt4/detection_loss‚3.072821329379904 &loss/total_loss‚41.817634080297424°ûuºhÝ*â 0loss/FRt1/assignment_loss‚1.2263612659251104 0loss/FRt3/assignment_loss‚3.2962008251084223 2loss/FBt3/assignment_loss‚0.002217652130610759 1loss/SRqqt1/detection_loss‚0.8470469637796646 trainer/global_step‚10999 _step‚906 /loss/FRt4/assignment_loss‚3.745804215299672 2loss/SRqqt3/assignment_loss‚1.2299031074035842 /loss/FRt1/detection_loss‚1.4048868778360184 1loss/SRqqt2/detection_loss‚2.2806296262048904 /loss/FBt3/detection_loss‚3.3534625470638275 2loss/SRqqt4/assignment_loss‚1.5541574967944103 .loss/FRt2/detection_loss‚2.343462366004323 /loss/FBt1/detection_loss‚0.6878256877340997 2loss/SRqqt1/assignment_loss‚0.7842284798843604 1loss/SRqqt2/assignment_loss‚1.285939568894044 2loss/FBt4/assignment_loss‚0.037327878869421384 .loss/FRt4/detection_loss‚3.072821329379904 &loss/total_loss‚41.817634080297424 /loss/FBt2/assignment_loss‚0.553244425507728 /loss/FRt3/detection_loss‚2.3630452348126307 /loss/FBt4/detection_loss‚2.2749447524547577 _runtime‚11107.878816882 0loss/FRt2/assignment_loss‚3.0862989702890085 1loss/SRqqt3/detection_loss‚1.6832430986863263 1loss/SRqqt4/detection_loss‚2.6073945294255796 /loss/FBt2/detection_loss‚2.0797194719314573 epoch‚9 2loss/FBt1/assignment_loss‚0.017467708877567212 % _timestamp‚1.7884391295235665e+09Š ‹{œÞ*‚(:‘ àÔåÔΨ²¶ gpu.0.gpu‚68.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚288.917' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.78339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y6óxß*‚(:í àÔåÔŠ˜ï· disk.sda2.out‚0$ memory_percent‚99.00257587432861# proc.memory.rssMB‚ 2862.84765625 disk.drbd2265.out‚0" disk.nvme0c0n1.in‚ 471.90234375( disk.nvme0c0n1.out‚81890.48681640625# disk.dm-0.out‚81890.48681640625 disk.sda3.in‚0.203125 disk.sda3.out‚ 414.11328125( proc.memory.availableMB‚ 163.41796875 disk./.usageGB‚ 287.900390625 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.473435401916504 network.recv‚ 3.78427e+06 disk.sda.in‚0.203125) disk./.usagePercent‚19.32123677936132 disk.sda.out‚ 414.11328125 disk.rbd1.out‚ 11.056640625 network.sent‚ 2.8357842e+07& disk.nvme0n1.out‚81890.48681640625 disk.dm-1.in‚0.203125 proc.cpu.threads‚17 disk.dm-0.in‚ 471.90234375 disk.sda1.in‚0 cpu‚25.288683673980792 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 414.11328125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375UB“à*‚(:ˆ ïÔåÔ‹ìȵ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚134.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã*{á*‚(:ð ïÔåÔ䯡¹ proc.cpu.threads‚17 network.sent‚ 2.8466729e+07 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 cpu‚25.288356189581403 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚81945.78955078125# disk.dm-0.out‚81945.78955078125 disk.sda3.in‚0.203125 disk.sda1.out‚0 network.recv‚ 3.790779e+06% disk./.usageGB‚287.91686248779297 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 414.76953125 disk.sda.out‚ 414.76953125" proc.memory.rssMB‚ 2865.5234375) proc.memory.percent‚17.48976707458496 disk.rbd1.out‚ 11.150390625 disk.dm-0.in‚ 471.90234375 disk.drbd2080.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 160.7265625) disk./.usagePercent‚19.32234221989417 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 414.76953125& disk.nvme0n1.out‚81945.78955078125$ memory_percent‚99.01900291442871–Ã=žâ*‚(:“ þÔåÔÁ™µ gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚195.164' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.032799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wþœçuã*‚(:ê þÔåÔÆ«·¶ disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.out‚ 11.212890625! proc.memory.rssMB‚ 2868.390625" disk.nvme0c0n1.in‚ 471.90234375# disk.dm-0.out‚82003.16064453125 disk.sda.in‚0.203125 disk.rbd0.in‚0& disk.nvme0n1.out‚82003.16064453125 cpu‚25.28802710315093 network.recv‚ 3.794574e+06% disk./.usageGB‚287.94152069091797 disk.sda3.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.sent‚ 2.847257e+07( disk.nvme0c0n1.out‚82003.16064453125 disk.sda3.out‚ 415.296875 disk.drbd2080.in‚0* disk./.usagePercent‚19.323997052595498 disk.drbd1716.in‚0 disk.sda.out‚ 415.296875 disk.dm-1.out‚ 415.296875* proc.memory.percent‚17.507266998291016 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0$ memory_percent‚99.03542995452881 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 158.03515625 disk.dm-0.in‚ 471.90234375 disk.sda1.out‚0 disk.drbd1716.out‚0(PçÎ4ä*/ _wandbruntime‚11153  _runtime‚11153áAq^Çå*Á % lr-AdamW/pg1‚4.58537215293688e-06 % lr-AdamW/pg2‚4.58537215293688e-06 trainer/global_step‚11049 % _timestamp‚1.7884391756175935e+09 _runtime‚11153.972645347 _step‚907vÝa‹Ìæ*Æ % lr-AdamW/pg2‚4.58537215293688e-06 trainer/global_step‚11049 % _timestamp‚1.7884391756175935e+09 _runtime‚11153.972645347 _step‚907 % lr-AdamW/pg1‚4.58537215293688e-06‹ÜBÕK4ç*/ _wandbruntime‚11153  _runtime‚11153ã$¼@#(è*P dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32756606898688 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1527475021499392 _runtime‚11153.97360625 _step‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238157003 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23624891995136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436660692 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205743272 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118779112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚11049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1503850155895808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238180109 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23624891995136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238157003 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118779115 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1503850155895808 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1284880544204288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436637580 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23588021911667 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1527478153284608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1501631834223731 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23627997388800 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23584918442743 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86964681 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86964160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1501628705326839 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23624891995136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238157003 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118779115 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1503850155895808 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1284880544204288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436637580 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23588021911667 XGDO~‡IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1527478153284608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1501631834223731 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23627997388800 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23584918442743 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86964681 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86964160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1501628705326839 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1527475021499392 _runtime‚11153.97360625 _step‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238157003 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23624891995136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436660692 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205743272 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118779112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚11049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1503850155895808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238180109 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205760957 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1317752534448128 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1284992955068416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198497867 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788439176510578e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚492 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23587027235843 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1503981649984000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86971592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32759498341376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1501760211121155 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118789368 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198497861 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205761452 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1317752453409792 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23630109715968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238178093 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1527611759699968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1527608627901440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436699092 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86972084 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118789365 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238178093 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1503981623592448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436699092 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238201225 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1503981649984000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1478173209313280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚11049 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ö ž2'ì*¬N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1284992983202816 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1478173183885312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436675954 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238201225 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1501763340018255 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23590130704975 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788439176510578e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚492 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23587027235843 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1503981649984000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86971592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32759498341376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1501760211121155 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118789368 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198497861 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205761452 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1317752453409792 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23630109715968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238178093 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1527611759699968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1527608627901440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436699092 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86972084 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118789365 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238178093 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1503981623592448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436699092 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238201225 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1503981649984000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1478173209313280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚11049 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23630109715968 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205760957 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1317752534448128 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1284992955068416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198497867 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _step‚912 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239237961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1484513636944896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23730522101760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239237961 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23733627495424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1534155356015616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1508204008633847 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚11099 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199344777 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119291762 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1510424833913856 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206635748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1534158487800832 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1510424833913856 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1510424860305408 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1290501087410688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438559632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1508207137530739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1534158487800832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23730522101760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438582744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199344783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32903758544896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23733627495424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119291765 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚87343447 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚32903703530496 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199344783 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1534155356015616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199344777 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23693475157875 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 $ _timestamp‚1.788439223030453e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239214855 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206635209 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1484513662372864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23690371688951 _runtime‚11201.385728341 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚87343983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440Øâßü((û*¢P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119291765 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚87343447 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚32903703530496 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199344783 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1534155356015616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199344777 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23693475157875 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 $ _timestamp‚1.788439223030453e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239214855 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206635209 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1484513662372864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23690371688951 _runtime‚11201.385728341 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚87343983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239214855 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 _step‚912 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239237961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1484513636944896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23730522101760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239237961 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23733627495424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1534155356015616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1508204008633847 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚11099 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199344777 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119291762 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1510424833913856 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206635748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1534158487800832 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1510424833913856 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1510424860305408 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1290501087410688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438559632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1508207137530739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1534158487800832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23730522101760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438582744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199344783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚32903758544896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23733627495424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329çåÑ–4ü*/ _wandbruntime‚11202  _runtime‚11202€ ÕÑ£&ý*¨N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239259077 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438621144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206653024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚913 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚11099 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199362067 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚32906680476672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 % _timestamp‚1.7884392239269865e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 438621144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199362061 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239259077 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1534292094216192 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438598006 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1290613498274816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1323520123750400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1508338643325263 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1510556354393600 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1510556328002048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚87351009 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1534288962417664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23735739822592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32906625475584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23692480482051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 C=Devi냊‹ceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971LºO3'þ*­N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23692480482051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239259077 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438621144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206653024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 _step‚913 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 trainer/global_step‚11099 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199362067 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚32906680476672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 % _timestamp‚1.7884392239269865e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 438621144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199362061 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239259077 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1534292094216192 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438598006 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1290613498274816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1323520123750400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1508338643325263 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1510556354393600 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1510556328002048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚87351009 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1534288962417664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23735739822592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚32906625475584‘÷ã4ÿ*/ _wandbruntime‚11202  _runtime‚11202''–6_€+Ù 2loss/SRqqt2/assignment_loss‚0.8900021321716762 0loss/SRqqt1/detection_loss‚0.800477113334402 1loss/SRqqt4/detection_loss‚2.2499509399587456 %loss/total_loss‚38.35171251612404 epoch‚9 _runtime‚11202.283911982 1loss/SRqqt1/assignment_loss‚1.013075941617811 2loss/SRqqt4/assignment_loss‚1.6776968933867686 3loss/FBt1/assignment_loss‚0.0007949013781727905 2loss/FBt2/assignment_loss‚0.013202806231674913 0loss/SRqqt2/detection_loss‚2.720311936878023 1loss/SRqqt3/detection_loss‚1.7414939684026383 /loss/FRt3/assignment_loss‚2.640464050769806 2loss/SRqqt3/assignment_loss‚0.8989829005052646 .loss/FBt2/detection_loss‚2.585366167128086 _step‚914 0loss/FRt1/assignment_loss‚1.3846878079267648 trainer/global_step‚11099 /loss/FRt4/detection_loss‚2.9509647137978496 $ _timestamp‚1.788439223928932e+09 /loss/FBt3/detection_loss‚2.2044160962104797 /loss/FRt1/detection_loss‚1.4626755998398249 /loss/FRt2/detection_loss‚2.2528444528579734 0loss/FRt2/assignment_loss‚2.8693020848127513 1loss/FBt4/assignment_loss‚0.17705871877671384 .loss/FRt4/assignment_loss‚2.93251895904541 1loss/FBt3/assignment_loss‚0.13780613106956302 .loss/FRt3/detection_loss‚2.336772049665451 /loss/FBt1/detection_loss‚0.5552506037479774 /loss/FBt4/detection_loss‚1.8555955466102152ølŒd+Þ .loss/FRt4/assignment_loss‚2.93251895904541 1loss/FBt3/assignment_loss‚0.13780613106956302 .loss/FRt3/detection_loss‚2.336772049665451 /loss/FBt1/detection_loss‚0.5552506037479774 /loss/FBt4/detection_loss‚1.8555955466102152 2loss/SRqqt2/assignment_loss‚0.8900021321716762 0loss/SRqqt1/detection_loss‚0.800477113334402 1loss/SRqqt4/detection_loss‚2.2499509399587456 %loss/total_loss‚38.35171251612404 epoch‚9 _runtime‚11202.283911982 1loss/SRqqt1/assignment_loss‚1.013075941617811 2loss/SRqqt4/assignment_loss‚1.6776968933867686 3loss/FBt1/assignment_loss‚0.0007949013781727905 2loss/FBt2/assignment_loss‚0.013202806231674913 0loss/SRqqt2/detection_loss‚2.720311936878023 1loss/SRqqt3/detection_loss‚1.7414939684026383 /loss/FRt3/assignment_loss‚2.640464050769806 2loss/SRqqt3/assignment_loss‚0.8989829005052646 .loss/FBt2/detection_loss‚2.585366167128086 _step‚914 0loss/FRt1/assignment_loss‚1.3846878079267648 trainer/global_step‚11099 /loss/FRt4/detection_loss‚2.9509647137978496 $ _timestamp‚1.788439223928932e+09 /loss/FBt3/detection_loss‚2.2044160962104797 /loss/FRt1/detection_loss‚1.4626755998398249 /loss/FRt2/detection_loss‚2.2528444528579734 0loss/FRt2/assignment_loss‚2.8693020848127513 1loss/FBt4/assignment_loss‚0.17705871877671384’SG}µ‚+‚(:’ ºÕåÔôøȵ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.27680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0݉“{ƒ+‚(:ð ºÕåÔ’àó¸% disk.nvme0n1.out‚82208.5556640625 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.sda.in‚0.203125$ memory_percent‚99.08773899078369 proc.cpu.threads‚17* disk./.usagePercent‚19.326196157314623 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.557764053344727 cpu‚25.286699299389596$ disk./.usageGB‚287.9742889404297 disk.rbd1.out‚ 11.5810546875' disk.nvme0c0n1.out‚82208.5556640625 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2876.6640625 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 149.46484375 disk.sda2.in‚0 network.sent‚ 2.8598202e+07 network.recv‚ 3.812352e+06 disk.rbd0.in‚0 disk.sda.out‚ 417.48046875 disk.dm-1.out‚ 417.48046875" disk.nvme0c0n1.in‚ 471.90234375" disk.dm-0.out‚82208.5556640625 disk.sda3.out‚ 417.48046875jýgΞ„+‚(:“ ÉÕåÔ¬«ÿ´ gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.967' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.993399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­vLH}…+‚(:ò ÉÕåÔÝÀ߶ disk.rbd1.out‚ 11.6748046875% disk./.usageGB‚288.00415420532227 disk.drbd1716.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 146.26953125" proc.memory.rssMB‚ 2880.1015625 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚82257.47607421875# disk.dm-0.out‚82257.47607421875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1716.in‚0 network.sent‚ 2.8707383e+07 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 418.08203125 disk.sda3.out‚ 418.08203125 cpu‚25.286371485036643 disk.sda.out‚ 418.08203125) disk./.usagePercent‚19.32820044030019 disk.drbd2080.out‚0 disk.rbd1.in‚0# memory_percent‚99.1072416305542& disk.nvme0n1.out‚82257.47607421875 disk.sda2.in‚0 disk.dm-0.in‚ 471.90234375 network.recv‚ 3.819068e+06 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375" disk.nvme0c0n1.in‚ 471.90234375* proc.memory.percent‚17.578744888305664 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0»Hõ††+‚(:’ ØÕåÔ×Üæ´ gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚287.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.510600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Úßw‡+‚(:ì ØÕåÔݾ· disk.sda3.out‚ 418.515625 disk.dm-0.in‚ 471.90234375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.in‚0 proc.cpu.threads‚17 network.sent‚ 2.8712718e+07 network.recv‚ 3.822592e+06 disk.sda1.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 11.7373046875 disk.rbd0.out‚0 disk.dm-1.out‚ 418.515625* disk./.usagePercent‚19.328387837815253$ disk./.usageGB‚288.0069465637207" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.drbd2265.in‚0( proc.memory.availableMB‚ 144.49609375 disk.sda.in‚0.203125 disk.sda2.in‚0$ memory_percent‚99.11806583404541 disk.drbd2080.out‚0 disk.drbd2265.out‚0* proc.memory.percent‚17.588090896606445# disk.dm-0.out‚82309.12451171875 disk.drbd2080.in‚0 disk.sda.out‚ 418.515625& disk.nvme0n1.out‚82309.12451171875 disk.sda2.out‚0" proc.memory.rssMB‚ 2881.6328125 cpu‚25.28607543196136 disk.dm-1.in‚0.203125 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚82309.12451171875(Œü4ˆ+/ _wandbruntime‚11248  _runtime‚11248z’ɉ+à & lr-AdamW/pg1‚3.751899114435485e-06 & lr-AdamW/pg2‚3.751899114435485e-06 trainer/global_step‚11149 % _timestamp‚1.7884392704282477e+09 _runtime‚11248.784005437 _step‚915RU³ÎŠ+È & lr-AdamW/pg1‚3.751899114435485e-06 & lr-AdamW/pg2‚3.751899114435485e-06 trainer/global_step‚11149 % _timestamp‚1.7884392704282477e+09 _runtime‚11248.784005437 _step‚915“Üž)y4‹+/ _wandbruntime‚11248  _runtime‚11248QC#(Œ+P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1516999538323456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1329172501808640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 440481684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119804412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1514782440837747 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 $ _timestamp‚1.788439270429213e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1296121658751488 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490983512433664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 200208977 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23795824935159 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1329172582860288 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23839257602048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚87725782 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _runtime‚11248.784678468 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1540835690531840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200208983 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚11149 _step‚916 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚87725782 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _runtime‚11248.784678468 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1540835690531840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200208983 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 trainer/global_step‚11149 _step‚916 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207547707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1517131032411648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240316929 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33053865507328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚87733539 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚11249.68710112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1517131032411648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240293797 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207548207 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1514910817735171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23797933728259 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1517131058803200 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1296234041481216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1329287935122944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240316929 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200226261 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33053812603392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23841369929216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491112884006912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240293797 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1514913946632271 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491112909434880 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440543196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440543196 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878£MPU2'+¬N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1517131032411648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240316929 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33053865507328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚87733539 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 _runtime‚11249.68710112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1517131032411648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240293797 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207548207 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1514910817735171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23797933728259 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1517131058803200 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1296234041481216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1329287935122944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240316929 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200226261 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33053812603392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23841369929216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491112884006912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240293797 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1514913946632271 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491112909434880 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440543196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440543196 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207547707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440••NN,4‘+/ _wandbruntime‚11249  _runtime‚11249‚<Á^’+Ø 2loss/SRqqt4/assignment_loss‚1.6938324988849702 /loss/FRt1/detection_loss‚1.4967652507540252 .loss/FRt2/detection_loss‚2.576873059208329 1loss/SRqqt1/detection_loss‚0.7472167483457689 0loss/FBt3/assignment_loss‚0.5919829197427987 /loss/FBt1/detection_loss‚0.5265640182420611 % _timestamp‚1.7884392713333328e+09 _runtime‚11249.688294833 2loss/SRqqt1/assignment_loss‚0.9777117704742411 epoch‚9 _step‚918 /loss/FRt1/assignment_loss‚1.375383984711435 0loss/FRt2/assignment_loss‚2.9615064279453174 2loss/FBt1/assignment_loss‚0.004344453797173843 .loss/FRt3/detection_loss‚2.425990648175541 .loss/FRt4/detection_loss‚3.364538872241974 1loss/SRqqt2/detection_loss‚2.4190799995608954 1loss/SRqqt4/detection_loss‚2.3494077517140295 /loss/FBt2/detection_loss‚2.8065763831138613 .loss/FBt3/detection_loss‚2.273372483253479 &loss/total_loss‚41.578654070892256 1loss/FBt4/assignment_loss‚0.08015375370159745 /loss/FRt4/assignment_loss‚3.584552252292633 2loss/SRqqt3/assignment_loss‚1.4895117445764217 0loss/FBt2/assignment_loss‚0.5708201096858829 1loss/SRqqt3/detection_loss‚1.5118542634628036 trainer/global_step‚11149 /loss/FRt3/assignment_loss‚2.514528175717906 2loss/SRqqt2/assignment_loss‚1.5337004240440286 /loss/FBt4/detection_loss‚1.7023860772450765^šDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1547516025048064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23941782315008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1521354615247863 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1523574242733056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 trainer/global_step‚11199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201073177 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523574269124608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201073177 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1497453362494464 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33198249397760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241330559 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884393175139103e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 442403736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23904381650291 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23901278181367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1523574269124608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241353665 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120317065 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1301742201957888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1547519156833280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1334940372401152 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201073183 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208424896 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1301742173823488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23944887708672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302áM )(¡+£P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23941782315008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1521354615247863 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1523574242733056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 trainer/global_step‚11199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50820096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201073177 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523574269124608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201073177 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1497453362494464 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33198249397760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241330559 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884393175139103e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 442403736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23904381650291 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23901278181367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1523574269124608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241353665 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120317065 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1301742201957888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1547519156833280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1334940372401152 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201073183 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208424896 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1301742173823488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23944887708672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚88108348 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33198198577664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1523574242733056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208425413 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1521357744144755 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1334940451355648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1547519156833280 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78954496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚11295.869131434 _step‚920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1547516025048064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864˜^c® 4¢+/ _wandbruntime‚11296  _runtime‚11296pŒV’_#£+§N YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1335055729526272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 442442110 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241374781 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1547652763248640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚88115225 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23906490443599 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1523705736821248 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1301854584687616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1523705763212800 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523705763212800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33201144838656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 442442110 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1301854612822016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23903386974467 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201090467 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241351649 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201090461 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241351649 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208442540 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚921 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1547649631450112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241374781 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884393184072235e+09 _runtime‚11296.76300215 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149758275949568‘°œÎ0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23943894628864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1547652763248640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 442465248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208443024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1497582734067712³~È@2'¤+¬N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23943894628864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1547652763248640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 442465248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208443024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1497582734067712 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1335055729526272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 442442110 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241374781 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1547652763248640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚88115225 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23906490443599 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1523705736821248 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1301854584687616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1523705763212800 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523705763212800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33201144838656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 442442110 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1301854612822016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23903386974467 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201090467 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241351649 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201090461 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241351649 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208442540 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚921 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1547649631450112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241374781 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 % _timestamp‚1.7884393184072235e+09 _runtime‚11296.76300215 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78941184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1497582759495680™d³…4¥+/ _wandbruntime‚11296  _runtime‚11296^³]¦+× %loss/total_loss‚41.73519320103243 trainer/global_step‚11199 /loss/FRt1/assignment_loss‚1.231560472647349 0loss/FRt2/assignment_loss‚3.1072916709459744 /loss/FRt2/detection_loss‚2.4099893218431716 /loss/FBt1/detection_loss‚0.6938029579961529 /loss/FRt4/assignment_loss‚4.073145077778743 /loss/FRt3/assignment_loss‚3.039731867429687 2loss/SRqqt2/assignment_loss‚1.8635281352326274 2loss/SRqqt3/assignment_loss‚0.9237959247496393 2loss/SRqqt4/assignment_loss‚1.4045948111168989 /loss/FBt2/detection_loss‚2.3656869903206825 _runtime‚11296.764478036 0loss/FBt3/assignment_loss‚0.4987116765728893 /loss/FRt1/detection_loss‚1.3341802273939054 /loss/FRt3/detection_loss‚2.5070879619296003 .loss/FRt4/detection_loss‚3.170599015859457 /loss/FBt3/detection_loss‚2.6301265822516546 .loss/FBt4/detection_loss‚2.072935771942139 epoch‚9 $ _timestamp‚1.788439318409515e+09 1loss/FBt1/assignment_loss‚0.01333879698960203 1loss/FBt2/assignment_loss‚0.29584465015796013 _step‚922 1loss/SRqqt1/detection_loss‚0.7655911236217148 0loss/SRqqt2/detection_loss‚2.357678313739598 2loss/SRqqt1/assignment_loss‚0.9414525451922887 1loss/FBt4/assignment_loss‚0.14248872138559818 1loss/SRqqt3/detection_loss‚1.8043296072218153 1loss/SRqqt4/detection_loss‚2.0877009767132835®Pb§+Ü /loss/FRt3/assignment_loss‚3.039731867429687 2loss/SRqqt2/assignment_loss‚1.8635281352326274 2loss/SRqqt3/assignment_loss‚0.9237959247496393 2loss/SRqqt4/assignment_loss‚1.4045948111168989 /loss/FBt2/detection_loss‚2.3656869903206825 _runtime‚11296.764478036 0loss/FBt3/assignment_loss‚0.4987116765728893 /loss/FRt1/detection_loss‚1.3341802273939054 /loss/FRt3/detection_loss‚2.5070879619296003 .loss/FRt4/detection_loss‚3.170599015859457 /loss/FBt3/detection_loss‚2.6301265822516546 .loss/FBt4/detection_loss‚2.072935771942139 epoch‚9 $ _timestamp‚1.788439318409515e+09 1loss/FBt1/assignment_loss‚0.01333879698960203 1loss/FBt2/assignment_loss‚0.29584465015796013 _step‚922 1loss/SRqqt1/detection_loss‚0.7655911236217148 0loss/SRqqt2/detection_loss‚2.357678313739598 2loss/SRqqt1/assignment_loss‚0.9414525451922887 1loss/FBt4/assignment_loss‚0.14248872138559818 1loss/SRqqt3/detection_loss‚1.8043296072218153 1loss/SRqqt4/detection_loss‚2.0877009767132835 %loss/total_loss‚41.73519320103243 trainer/global_step‚11199 /loss/FRt1/assignment_loss‚1.231560472647349 0loss/FRt2/assignment_loss‚3.1072916709459744 /loss/FRt2/detection_loss‚2.4099893218431716 /loss/FBt1/detection_loss‚0.6938029579961529 /loss/FRt4/assignment_loss‚4.073145077778743šÓ¼£š“¨+‚(:ˆ £ÖåÔ¦ª…µ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚297.097' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o8(¹{©+‚(:ð £ÖåÔÁâÛ¶ disk.sda.out‚ 421.26171875 network.recv‚ 3.845772e+06* disk./.usagePercent‚19.343279540074477 disk.rbd1.out‚ 12.01953125 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 cpu‚25.28447290582589& disk.nvme0n1.out‚82672.80615234375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2885.53515625( disk.nvme0c0n1.out‚82672.80615234375 proc.cpu.threads‚17 disk.dm-1.out‚ 421.26171875 disk.sda.in‚0.203125# memory_percent‚99.1377592086792) proc.memory.percent‚17.61190891265869 disk.sda3.out‚ 421.26171875 disk.sda1.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 141.26953125 network.sent‚ 2.8845274e+07 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.in‚0.203125 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0% disk./.usageGB‚288.22884368896484 disk.drbd2266.in‚0# disk.dm-0.out‚82672.80615234375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0Cœª+‚(:’ ²ÖåÔµËƵ gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚270.533' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.10660000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º&ÇŽr«+‚(:ç ²ÖåÔÂÁí¶ network.sent‚ 2.8954156e+07 network.recv‚ 3.851889e+06 disk.rbd0.in‚0 disk.sda.in‚0.203125 disk.dm-0.in‚ 471.90234375" disk.dm-0.out‚82728.5966796875 disk.sda1.out‚0) disk./.usagePercent‚19.34693174355795 disk.drbd2266.out‚0 disk.rbd1.out‚ 12.08203125 disk.drbd1716.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚17.619657516479492% disk./.usageGB‚288.28326416015625% disk.nvme0n1.out‚82728.5966796875" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125( proc.memory.availableMB‚ 140.00390625 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0' disk.nvme0c0n1.out‚82728.5966796875" proc.memory.rssMB‚ 2886.8046875 proc.cpu.threads‚17 disk.sda3.out‚ 421.734375$ memory_percent‚99.14548397064209 disk.sda2.out‚0 disk.dm-1.out‚ 421.734375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.out‚ 421.734375 cpu‚25.28416239621532 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2080.in‚0Åù|%¬+‚(:’ ÁÖåÔ éà´ gpu.0.gpu‚90.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚283.153' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.63060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_J”Ðs­+‚(:è ÁÖåÔ郱·( disk.nvme0c0n1.out‚82791.75537109375 disk.drbd2080.in‚0 disk.drbd2265.out‚0$ memory_percent‚99.15204048156738 disk.sda2.out‚0# disk.dm-0.out‚82791.75537109375 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.sda.in‚0.203125# proc.memory.rssMB‚ 2887.76171875 cpu‚25.283850963139496 disk.rbd1.in‚0 disk.rbd1.out‚ 12.08203125& disk.nvme0n1.out‚82791.75537109375 disk.sda2.in‚0' proc.memory.availableMB‚ 138.9296875 proc.cpu.threads‚17* disk./.usagePercent‚19.349554028731173 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.recv‚ 3.856869e+06 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.out‚ 422.53125" disk.nvme0c0n1.in‚ 471.90234375 disk.rbd0.out‚0 disk.dm-0.in‚ 471.90234375 disk.drbd2080.out‚0 network.sent‚ 2.8960605e+07% disk./.usageGB‚288.32233810424805 disk.drbd2266.out‚0 disk.sda3.out‚ 422.53125 disk.sda.out‚ 422.53125 disk.drbd2265.in‚0 disk.dm-1.in‚0.203125 disk.drbd1716.out‚0) proc.memory.percent‚17.62549877166748Í´½4®+/ _wandbruntime‚11342  _runtime‚11342—Öɯ+à _runtime‚11342.882044766 _step‚923 & lr-AdamW/pg1‚3.000558576603908e-06 & lr-AdamW/pg2‚3.000558576603908e-06 trainer/global_step‚11249 % _timestamp‚1.7884393645270455e+09»^¨lΰ+È trainer/global_step‚11249 % _timestamp‚1.7884393645270455e+09 _runtime‚11342.882044766 _step‚923 & lr-AdamW/pg1‚3.000558576603908e-06 & lr-AdamW/pg2‚3.000558576603908e-06›Ÿ)T4±+/ _wandbruntime‚11342  _runtime‚11342¤$(²+žP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1527929918554871 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24047412421632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33345131502592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242411517 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201937377 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209316074 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24009834896499 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201937383 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120829712 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1503923187127296 trainer/global_step‚11249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24050517815296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1530148947142656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24050517815296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209315535 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201937377 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1530148947142656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201937383 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1307362745164288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1530148973534208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1503923212555264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 444325788 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1554199491349504 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1554196359564288 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1530148973534208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1340707793518080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1527933047451763 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚88485823 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1554199491349504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242388411 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242388411 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚88486359 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24047412421632 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33345076488192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24006731427575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1307362717029888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884393645277312e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1340707876666880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242411517 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 444348900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1554196359564288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032å®uwF ³+£P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1530148973534208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1503923212555264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 444325788 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1554199491349504 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1554196359564288 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1530148973534208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1340707793518080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1527933047451763 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚88485823 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1554199491349504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_poolJ‹óã.freed‚ 242388411 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242388411 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚88486359 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24047412421632 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33345076488192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24006731427575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1307362717029888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884393645277312e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1340707876666880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242411517 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 444348900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1554196359564288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1527929918554871 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24047412421632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33345131502592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242411517 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201937377 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209316074 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24009834896499 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201937383 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120829712 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1503923187127296 trainer/global_step‚11249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24050517815296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1530148947142656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24050517815296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209315535 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201937377 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1530148947142656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201937383 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1307362745164288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 444387300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1504052609556480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11343.762734992 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1554333097764864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33348072431104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1530280467622400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24049524735488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1307475127894016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120839968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201954661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚925 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201954667 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209333284 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24008840220675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201954661 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚88493821 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1307475156028416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1554333097764864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1554329965966336 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1528064553246287 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24011943689807 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1530280467622400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚88493319 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242409501 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 % _timestamp‚1.7884393654070277e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 444387300 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 242432633 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24052630142464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24052630142464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 242432633›¶Í3'¶+­N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 444387300 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 242432633 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24052630142464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24052630142464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 242432633 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1340823228459520 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 444364162 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 444387300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1504052609556480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11343.762734992 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1554333097764864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚81038336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33348072431104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1530280467622400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24049524735488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1307475127894016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120839968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201954661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚925 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201954667 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209333284 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24008840220675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201954661 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚88493821 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1307475156028416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1554333097764864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1554329965966336 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1528064553246287 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24011943689807 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1530280467622400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚88493319 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242409501 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 % _timestamp‚1.7884393654070277e+09<ám4·+/ _wandbruntime‚11343  _runtime‚11343N°>èc¸+Ý 0loss/FBt2/assignment_loss‚0.6124863404159737 /loss/FRt3/detection_loss‚2.5315515852787276 0loss/SRqqt4/detection_loss‚2.632411422552886 0loss/FRt1/assignment_loss‚1.1936324593183156 2loss/SRqqt3/assignment_loss‚1.0743284222226954 1loss/FBt3/assignment_loss‚0.19637219961282426 % _timestamp‚1.7884393654089127e+09 _runtime‚11343.763852501 /loss/FBt1/detection_loss‚0.5628363182629222 trainer/global_step‚11249 /loss/FRt3/assignment_loss‚2.165067646991123 2loss/SRqqt1/assignment_loss‚1.0801196026851434 4loss/FBt1/assignment_loss‚0.00012559476821829164 1loss/SRqqt1/detection_loss‚0.8380208357001644 /loss/FRt4/assignment_loss‚2.370463564282372 _step‚926 /loss/FRt2/assignment_loss‚3.050599384307861 2loss/SRqqt4/assignment_loss‚1.5704530927318114 /loss/FBt2/detection_loss‚2.6362635493278503 %loss/total_loss‚39.10251651938832 1loss/FBt4/assignment_loss‚0.15420051387189102 /loss/FRt2/detection_loss‚2.3907470464706444 /loss/FRt4/detection_loss‚3.3450733508382524 epoch‚9 2loss/SRqqt2/assignment_loss‚1.5924675493900264 /loss/FRt1/detection_loss‚1.1823194525330454 1loss/SRqqt2/detection_loss‚2.1202043209757124 1loss/SRqqt3/detection_loss‚1.8476457962902582 /loss/FBt3/detection_loss‚2.0237241875041616 /loss/FBt4/detection_loss‚1.9314022830554418Ý #h¹+â 2loss/SRqqt3/assignment_loss‚1.0743284222226954 1loss/FBt3/assignment_loss‚0.19637219961282426 % _timestamp‚1.7884393654089127e+09 _runtime‚11343.763852501 /loss/FBt1/detection_loss‚0.5628363182629222 trainer/global_step‚11249 /loss/FRt3/assignment_loss‚2.165067646991123 2loss/SRqqt1/assignment_loss‚1.0801196026851434 4loss/FBt1/assignment_loss‚0.00012559476821829164 1loss/SRqqt1/detection_loss‚0.8380208357001644 /loss/FRt4/assignment_loss‚2.370463564282372 _step‚926 /loss/FRt2/assignment_loss‚3.050599384307861 2loss/SRqqt4/assignment_loss‚1.5704530927318114 /loss/FBt2/detection_loss‚2.6362635493278503 %loss/total_loss‚39.10251651938832 1loss/FBt4/assignment_loss‚0.15420051387189102 /loss/FRt2/detection_loss‚2.3907470464706444 /loss/FRt4/detection_loss‚3.3450733508382524 epoch‚9 2loss/SRqqt2/assignment_loss‚1.5924675493900264 /loss/FRt1/detection_loss‚1.1823194525330454 1loss/SRqqt2/detection_loss‚2.1202043209757124 1loss/SRqqt3/detection_loss‚1.8476457962902582 /loss/FBt3/detection_loss‚2.0237241875041616 /loss/FBt4/detection_loss‚1.9314022830554418 0loss/FBt2/assignment_loss‚0.6124863404159737 /loss/FRt3/detection_loss‚2.5315515852787276 0loss/SRqqt4/detection_loss‚2.632411422552886 0loss/FRt1/assignment_loss‚1.1936324593183156žGæJhžº+‚(:“ ÐÖåÔ“Ìœµ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚293.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.684999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j :O{»+‚(:ð ÐÖåÔÑ‚Ù¶ network.sent‚ 2.8965895e+07 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2888.71484375 disk.drbd1716.in‚0 disk.sda.out‚ 423.07421875% disk./.usageGB‚288.36015701293945 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚82866.24365234375) proc.memory.percent‚17.63131618499756 network.recv‚ 3.860508e+06 disk.drbd2080.out‚0& disk.nvme0n1.out‚82866.24365234375 disk.rbd1.out‚ 12.1142578125 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚82866.24365234375$ memory_percent‚99.15733337402344$ proc.memory.availableMB‚138.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 423.07421875 disk.rbd1.in‚0 cpu‚25.283545869098667" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚0.203125 proc.cpu.threads‚17* disk./.usagePercent‚19.352092087425632 disk.sda3.out‚ 423.07421875%ríû’¼+‚(:‡ ßÖåÔø­¢µ gpu.0.gpu‚71.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.435' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.287 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G¤ÓÏ‘½+‚(:ê ßÖåÔÛζ" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd2266.in‚0$ memory_percent‚99.15962219238281* proc.memory.percent‚17.632293701171875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚82926.14892578125 disk.dm-0.in‚ 471.90234375 disk.drbd2265.in‚0 proc.memory.rssMB‚2888.875* disk./.usagePercent‚19.352611014697533 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 423.66015625 disk.drbd2265.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚82926.14892578125 disk.sda3.out‚ 423.66015625 disk.sda1.out‚0$ disk./.usageGB‚288.3678894042969 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 12.2080078125 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375# disk.dm-0.out‚<òí(ä82926.14892578125 disk.dm-1.in‚0.203125$ proc.memory.availableMB‚137.6875 cpu‚25.283217345193624 network.sent‚ 2.9074728e+07 network.recv‚ 3.86667e+06 disk.drbd2080.in‚0 disk.dm-1.out‚ 423.66015625ác“¾+‚(:ˆ îÖåÔäÆÕµ gpu.0.gpu‚67.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯2 {¿+‚(:ð îÖåÔÏÈ· disk.drbd2266.in‚0 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚19.35552566047486 disk.sda1.out‚0# disk./.usageGB‚288.411319732666 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 cpu‚25.28290593515287 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0 disk.dm-1.out‚ 424.41015625 disk.rbd1.in‚0 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚82984.48681640625# disk.dm-0.out‚82984.48681640625 disk.drbd1716.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2889.1953125 network.recv‚ 3.870605e+06 disk.sda2.out‚0 disk.sda3.out‚ 424.41015625* proc.memory.percent‚17.634248733520508 disk.dm-1.in‚0.203125 disk.drbd2080.out‚0 disk.sda.out‚ 424.41015625 disk.drbd2265.in‚0$ memory_percent‚99.16026592254639( proc.memory.availableMB‚ 137.58203125& disk.nvme0n1.out‚82984.48681640625 disk.rbd1.out‚ 12.2705078125 disk.dm-0.in‚ 471.90234375 disk.drbd2265.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.203125 network.sent‚ 2.9080679e+073¦uº4À+/ _wandbruntime‚11390  _runtime‚11390‘xÇËÁ+Å ' lr-AdamW/pg1‚2.6558863969921352e-06 ' lr-AdamW/pg2‚2.6558863969921352e-06 trainer/global_step‚11299 % _timestamp‚1.7884394116788802e+09 _runtime‚11390.033898638 _step‚927pY"ÐÂ+Ê % _timestamp‚1.7884394116788802e+09 _runtime‚11390.033898638 _step‚927 ' lr-AdamW/pg1‚2.6558863969921352e-06 ' lr-AdamW/pg2‚2.6558863969921352e-06 trainer/global_step‚11299ŸÇ´G?4Ã+/ _wandbruntime‚11390  _runtime‚11390#ŽÐ#(Ä+P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210206004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1536723651552256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 446270952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243469369 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243446263 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1536723677943808 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560876694080512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202801577 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24156147921920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1534505221861879 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1560879825865728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210205469 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33491778338816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24153042528256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚11299 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚81051648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1560879825865728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243446263 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560876694080512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1536723677943808 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1312983260236288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 446247840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 _runtime‚11390.03505511 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560876694080512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1536723677943808 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1312983260236288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 446247840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 _runtime‚11390.03505511 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210206004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1536723651552256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 446270952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243469369 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243446263 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1536723677943808 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560876694080512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202801577 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24156147921920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 _step‚928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 š…`k4Æ+/ _wandbruntime‚11390  _runtime‚11390PžºU?&Ç+¦N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚88871183 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1536855145640448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚88870666 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1561013432281088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210223801 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121352618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 446286214 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1534636727656195 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 202818867 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 202818861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1536855172032000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210223281 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1313095699234816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33494647499264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33494700403200 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1510522459617280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24158260249088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 446286214 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1313095671100416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202818861 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1534639856553295 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1510522434189312 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚11390.921485731 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 446309352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1536855172032000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1536855145640448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 446309352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.cT4t·íurrent‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243467353m!Å1'È+«N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1561013432281088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210223801 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121352618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 446286214 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1534636727656195 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 202818867 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 202818861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52903936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1536855172032000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210223281 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1313095699234816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33494647499264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33494700403200 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1510522459617280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24158260249088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 446286214 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1313095671100416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202818861 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1534639856553295 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1510522434189312 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 _runtime‚11390.921485731 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 446309352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1536855172032000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1536855145640448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 446309352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243467353 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚88871183 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1536855145640448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚88870666 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132¡}Ö —4É+/ _wandbruntime‚11390  _runtime‚11390ÐfEÉZÊ+Ô /loss/FRt1/detection_loss‚1.6799018658530802 0loss/FRt1/assignment_loss‚1.6898457452432434 /loss/FRt3/assignment_loss‚2.559755998413737 0loss/SRqqt2/assignment_loss‚1.64781736275729 .loss/FBt4/detection_loss‚2.245196466644605 $ _timestamp‚1.788439412567546e+09 %loss/total_loss‚41.62026172591305 trainer/global_step‚11299 _runtime‚11390.922511828 0loss/FBt3/assignment_loss‚0.2696907355632654 .loss/FRt2/detection_loss‚2.296604668221825 0loss/SRqqt2/detection_loss‚2.334029225742116 1loss/SRqqt3/detection_loss‚1.6357359904153594 1loss/SRqqt4/detection_loss‚2.2537753958912456 /loss/FBt1/detection_loss‚0.5689522765842931 2loss/SRqqt1/assignment_loss‚0.9240802808822146 2loss/SRqqt3/assignment_loss‚1.3860691314852187 2loss/FBt4/assignment_loss‚0.038886114673611395 0loss/SRqqt1/detection_loss‚1.051865301921334 /loss/FRt3/detection_loss‚2.2272191963544707 .loss/FBt3/detection_loss‚1.916500137610869 _step‚930 0loss/FRt2/assignment_loss‚3.2813337605173993 2loss/SRqqt4/assignment_loss‚1.8410168614457636 /loss/FRt4/detection_loss‚2.7280478436371376 /loss/FBt2/detection_loss‚2.8953614149774825 epoch‚9 /loss/FRt4/assignment_loss‚4.070037541718318 2loss/FBt1/assignment_loss‚0.003540447305842166 /loss/FBt2/assignment_loss‚0.074997962053333‹ë¹_Ë+Ù 0loss/FRt1/assignment_loss‚1.6898457452432434 /loss/FRt3/assignment_loss‚2.559755998413737 0loss/SRqqt2/assignment_loss‚1.64781736275729 .loss/FBt4/detection_loss‚2.245196466644605 $ _timestamp‚1.788439412567546e+09 %loss/total_loss‚41.62026172591305 trainer/global_step‚11299 _runtime‚11390.922511828 0loss/FBt3/assignment_loss‚0.2696907355632654 .loss/FRt2/detection_loss‚2.296604668221825 0loss/SRqqt2/detection_loss‚2.334029225742116 1loss/SRqqt3/detection_loss‚1.6357359904153594 1loss/SRqqt4/detection_loss‚2.2537753958912456 /loss/FBt1/detection_loss‚0.5689522765842931 2loss/SRqqt1/assignment_loss‚0.9240802808822146 2loss/SRqqt3/assignment_loss‚1.3860691314852187 2loss/FBt4/assignment_loss‚0.038886114673611395 0loss/SRqqt1/detection_loss‚1.051865301921334 /loss/FRt3/detection_loss‚2.2272191963544707 .loss/FBt3/detection_loss‚1.916500137610869 _step‚930 0loss/FRt2/assignment_loss‚3.2813337605173993 2loss/SRqqt4/assignment_loss‚1.8410168614457636 /loss/FRt4/detection_loss‚2.7280478436371376 /loss/FBt2/detection_loss‚2.8953614149774825 epoch‚9 /loss/FRt4/assignment_loss‚4.070037541718318 2loss/FBt1/assignment_loss‚0.003540447305842166 /loss/FBt2/assignment_loss‚0.074997962053333 /loss/FRt1/detection_loss‚1.6799018658530802¢õw/{“Ì+‚(:ˆ ýÖåÔñ›¶ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gnªYyÍ+‚(:î ýÖåÔý²˜· network.recv‚ 3.875602e+06 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 proc.cpu.threads‚17" disk.dm-0.out‚83040.8212890625$ memory_percent‚99.16355609893799 disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚25.282601880257797* disk./.usagePercent‚19.356462392042648 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2889.3515625' disk.nvme0c0n1.out‚83040.8212890625 disk.sda3.out‚ 424.9453125 disk.drbd2265.out‚0* proc.memory.percent‚17.635202407836914 network.sent‚ 2.9087082e+07% disk./.usageGB‚288.42527770996094 disk.dm-1.out‚ 424.9453125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375 disk.sda1.in‚0 disk.sda.out‚ 424.9453125% disk.nvme0n1.out‚83040.8212890625( proc.memory.availableMB‚ 137.04296875 disk.drbd2266.in‚0 disk.rbd1.out‚ 12.2705078125 disk.sda1.out‚0 disk.sda.in‚0.203125³l“Î+‚(:ˆ Œ×åÔÚŠ±¶ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚262.376' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.4752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h¿.dxÏ+‚(:í Œ×åÔ¸½Â¸# proc.memory.rssMB‚ 2889.66796875* proc.memory.percent‚17.637133598327637$ disk./.usageGB‚288.4555854797363 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 cpu‚25.282290847530206 network.sent‚ 2.9195309e+07 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 12.302734375 disk.sda1.in‚0" disk.dm-0.out‚83092.6611328125$ memory_percent‚99.16362762451172 network.recv‚ 3.881288e+06 disk.drbd2080.out‚0 disk.sda.out‚ 425.32421875 disk.nvme0n1.in‚ 471.90234375" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.out‚ 425.32421875 disk.sda.in‚0.203125% disk.nvme0n1.out‚83092.6611328125 disk.drbd2266.out‚0 disk.dm-0.in‚ 471.90234375* disk./.usagePercent‚19.358496371902188 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 425.32421875% proc.memory.availableMB‚ 137.03125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚83092.6611328125 disk.sda3.in‚0.203125 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0L®§“Ð+‚(:ˆ ›×åÔÈú´ gpu.0.gpu‚68.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²îò1mÑ+‚(:â ›×åÔëÃÅ· disk.dm-1.out‚ 426.0078125 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.out‚83149.78515625% proc.memory.availableMB‚ 136.21875# proc.memory.rssMB‚ 2889.98828125 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-0.in‚ 471.90234375 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 471.90234375# disk.nvme0n1.out‚83149.78515625 proc.cpu.threads‚17$ disk./.usageGB‚288.4988098144531% disk.nvme0c0n1.out‚83149.78515625 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚ 3.88574e+06 disk.rbd1.in‚0 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.drbd2265.in‚0* disk./.usagePercent‚19.361397193272662 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.63908863067627 cpu‚25.281969372450757 disk.sda2.in‚0 disk.rbd1.out‚ 12.302734375 disk.sda.out‚ 426.0078125 network.sent‚ 2.9201229e+07 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 426.0078125$ memory_percent‚99.16858673095703}oÙá4Ò+/ _wandbruntime‚11436  _runtime‚11436VßIÝËÓ+Å ' lr-AdamW/pg2‚2.3319768197792654e-06 trainer/global_step‚11349 % _timestamp‚1.7884394585952544e+09 _runtime‚11436.950285926 _step‚931 ' lr-AdamW/pg1‚2.3319768197792654e-06~ÂÐÔ+Ê ' lr-AdamW/pg1‚2.3319768197792654e-06 ' lr-AdamW/pg2‚2.3319768197792654e-06 trainer/global_step‚11349 % _timestamp‚1.7884394585952544e+09 _runtime‚11436.950285926 _step‚931£G¿òú4Õ+/ _wandbruntime‚11436  _runtime‚11436{B:Û#(Ö+P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203665777 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1567560160381952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203665783 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1543298355961856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33638247729664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 448169892 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1567557028596736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203665783 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244527221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24217637919991 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211096286 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211096834 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1567560160381952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33638192715264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 _step‚932 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244504115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1543298382353408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1352242079306752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1567557028596736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121855015 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121855012 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1352241996157952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24258672634880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1543298355961856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚89241274 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1318603831577088F“~ßA×+¢P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1567557028596736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203665783 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244527221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24217637919991 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211096286 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211096834 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1567560160381952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33638192715264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 _step‚932 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 24450411Ä2ª§ç5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1543298382353408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1352242079306752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1567557028596736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121855015 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121855012 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1352241996157952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24258672634880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1543298355961856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚89241274 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1318603831577088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203665777 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1567560160381952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203665783 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1543298355961856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33638247729664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 448169892 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚11349 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244548337 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244525205 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 448231404 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 448208266 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203683067 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121865268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 203683067 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33641202101248 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24263890355712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24260784948736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121865265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1516992309678080 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚89248845 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244548337 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211114630 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1352357361406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24263890355712 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33641147100160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1567690634998784 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚89249362 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211114110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1352357444542464 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 448231404 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203683061 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244525205GÔ´3'Ú+­N ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1352357444542464 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 448231404 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203683061 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244525205 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1541212030963203 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24222850182223 _runtime‚11437.837472582 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1543429876441600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1318716242441216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 % _timestamp‚1.7884394594819052e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1543429850050048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24260784948736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1567693766797312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 trainer/global_step‚11349 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244548337 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244525205 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 448231404 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 448208266 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203683067 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121865268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 203683067 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33641202101248 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24263890355712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24260784948736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121865265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1516992309678080 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚89248845 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244548337 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211114630 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1352357361406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24263890355712 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33641147100160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1567690634998784 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 _step‚933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚89249362 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211114110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3¥;¬x$4Û+/ _wandbruntime‚11437  _runtime‚11437ú¿rbÜ+Ü 1loss/SRqqt2/assignment_loss‚1.575429506599903 /loss/FRt1/detection_loss‚1.5311660118950683 $ _timestamp‚1.788439459483455e+09 _runtime‚11437.838386626 2loss/SRqqt1/assignment_loss‚0.8906754665194622 2loss/FBt3/assignment_loss‚0.004415779690508056 .loss/FBt2/detection_loss‚2.392727881669998 3loss/FBt1/assignment_loss‚0.0008507270773744671 1loss/FBt4/assignment_loss‚0.03444369163286562 0loss/SRqqt3/detection_loss‚1.595755354859508 1loss/SRqqt4/detection_loss‚2.4446086968694414 /loss/FBt3/detection_loss‚3.0736502153532848 _step‚934 0loss/FRt2/assignment_loss‚2.8354058235883715 1loss/SRqqt2/detection_loss‚2.0654543712735176 0loss/FBt1/detection_loss‚0.48931708079404557 2loss/SRqqt4/assignment_loss‚1.4895313759999615 .loss/FRt3/detection_loss‚2.498819251393163 .loss/FRt4/detection_loss‚2.805688397304432 /loss/FBt4/detection_loss‚2.1931112507979074 0loss/FRt4/assignment_loss‚3.0901459149412207 1loss/FBt2/assignment_loss‚0.43950142175890505 epoch‚9 trainer/global_step‚11349 2loss/SRqqt3/assignment_loss‚1.1515272615764003 1loss/SRqqt1/detection_loss‚1.0836512431785275 %loss/total_loss‚40.84578154069882 0loss/FRt1/assignment_loss‚1.1910322943671805 0loss/FRt3/assignment_loss‚3.5918696855389793 .loss/FRt2/detection_loss‚2.377002836018801¼çkgÝ+á 0loss/FRt2/assignment_loss‚2.8354058235883715 1loss/SRqqt2/detection_loss‚2.0654543712735176 0loss/FBt1/detection_loss‚0.48931708079404557 2loss/SRqqt4/assignment_loss‚1.4895313759999615 .loss/FRt3/detection_loss‚2.498819251393163 .loss/FRt4/detection_loss‚2.805688397304432 /loss/FBt4/detection_loss‚2.1931112507979074 0loss/FRt4/assignment_loss‚3.0901459149412207 1loss/FBt2/assignment_loss‚0.43950142175890505 epoch‚9 trainer/global_step‚11349 2loss/SRqqt3/assignment_loss‚1.1515272615764003 1loss/SRqqt1/detection_loss‚1.0836512431785275 %loss/total_loss‚40.84578154069882 0loss/FRt1/assignment_loss‚1.1910322943671805 0loss/FRt3/assignment_loss‚3.5918696855389793 .loss/FRt2/detection_loss‚2.377002836018801 1loss/SRqqt2/assignment_loss‚1.575429506599903 /loss/FRt1/detection_loss‚1.5311660118950683 $ _timestamp‚1.788439459483455e+09 _runtime‚11437.838386626 2loss/SRqqt1/assignment_loss‚0.8906754665194622 2loss/FBt3/assignment_loss‚0.004415779690508056 .loss/FBt2/detection_loss‚2.392727881669998 3loss/FBt1/assignment_loss‚0.0008507270773744671 1loss/FBt4/assignment_loss‚0.03444369163286562 0loss/SRqqt3/detection_loss‚1.595755354859508 1loss/SRqqt4/detection_loss‚2.4446086968694414 /loss/FBt3/detection_loss‚3.0736502153532848 _step‚934¦Dý“Þ+‚(:ˆ ª×åÔÍʵ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚284.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž“å™bß+‚(:× ª×åÔ¬‹ã¶( proc.memory.percent‚17.6419734954834 network.recv‚ 3.88985e+06 disk.drbd1716.out‚0 disk.sda.out‚ 426.59375 disk.drbd2265.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 136.01953125 network.sent‚ 2.92071e+07$ disk./.usageGB‚288.5287399291992 disk.dm-0.out‚ 83198.4609375 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2890.4609375 cpu‚25.28164773443209$ disk.nvme0c0n1.out‚ 83198.4609375* disk./.usagePercent‚19.363405828386313" disk.nvme0c0n1.in‚ 471.90234375 disk.sda.in‚0.203125" disk.nvme0n1.out‚ 83198.4609375 disk.sda2.in‚0$ memory_percent‚99.16980266571045 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 proc.cpu.threads‚17 disk.rbd1.out‚ 12.365234375 disk.sda3.out‚ 426.59375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 426.59375¡ã箓à+‚(:ˆ ¹×åÔ¯Àµµ gpu.0.gpu‚65.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚257.892' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÃ/yá+‚(:î ¹×åÔªö¶ disk.drbd2266.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0% disk./.usageGB‚288.56360626220703( disk.nvme0c0n1.out‚83260.38525390625 disk.dm-0.in‚ 471.90234375 disk.drbd2265.out‚0* proc.memory.percent‚17.645883560180664 proc.cpu.threads‚17 network.sent‚ 2.9316259e+07 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125# disk.dm-0.out‚83260.38525390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 471.90234375$ memory_percent‚99.17564392089844" proc.memory.rssMB‚ 2891.1015625 disk.sda2.out‚0 disk.dm-1.out‚ 427.32421875 disk.drbd1716.in‚0 disk.sda.in‚0.203125* disk./.usagePercent‚19.365745737249267 disk.sda3.out‚ 427.32421875 disk.drbd2080.in‚0$ proc.memory.availableMB‚135.0625 cpu‚25.28134450429577 network.recv‚ 3.896961e+06 disk.rbd1.out‚ 12.365234375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 427.32421875& disk.nvme0n1.out‚83260.38525390625î8“âžâ+‚(:“ È×åÔã–ã´ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚288.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.791199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l¦ã+‚(:è È×åÔ»Û’· disk.drbd2266.in‚0# disk.dm-0.out‚83319.41650390625 disk.drbd2080.in‚0$ memory_percent‚99.17562009A/0æ7904053* proc.memory.percent‚17.645883560180664 network.sent‚ 2.9320606e+07 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2891.1015625 disk.dm-0.in‚ 471.90234375 disk.sda3.out‚ 427.71875 disk.sda2.out‚0( proc.memory.availableMB‚ 135.06640625 cpu‚25.281020003162336 disk.rbd1.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0) disk./.usagePercent‚19.36764940927388 disk.rbd1.out‚ 12.380859375 disk.sda.out‚ 427.71875 disk.drbd2265.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚83319.41650390625 proc.cpu.threads‚17 network.recv‚ 3.899413e+06$ disk./.usageGB‚288.5919723510742( disk.nvme0c0n1.out‚83319.41650390625 disk.sda3.in‚0.203125 disk.sda.in‚0.203125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 427.71875" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.in‚0øoV4ä+/ _wandbruntime‚11483  _runtime‚11483LpêÊå+Ä ' lr-AdamW/pg1‚2.0288973472356793e-06 ' lr-AdamW/pg2‚2.0288973472356793e-06 trainer/global_step‚11399 $ _timestamp‚1.788439505303776e+09 _runtime‚11483.658859958 _step‚935“N—Ïæ+É ' lr-AdamW/pg2‚2.0288973472356793e-06 trainer/global_step‚11399 $ _timestamp‚1.788439505303776e+09 _runtime‚11483.658859958 _step‚935 ' lr-AdamW/pg1‚2.0288973472356793e-06§ }d4ç+/ _wandbruntime‚11483  _runtime‚11483“JH;$(è+žP [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24364302741504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204529983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1547658957372787 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24326194635123 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 450091944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204529977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245585073 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1549873086763008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _step‚936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1358008741315072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚11399 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1547655828475895 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122367665 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204529977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245561967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33784451777536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24364302741504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211987042 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1358008826561024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33784394665984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1324224374783488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1574240494898176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24323091166199 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1574237363112960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 450115056 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1574237363112960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24367408135168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1523332762737664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1549873060371456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24367408135168¸ç=)(é+£P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1574240494898176 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1574237363112960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24367408135168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1523332762737664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1549873060371456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24367408135168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24364302741504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204529983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1547658957372787 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24326194635123 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 450091944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204529977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245585073 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1549873086763008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 _step‚936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1358008741315072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 trainer/global_step‚11399 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1547655828475895 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122367665 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204529977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245561967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33784451777536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24364302741504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211987042 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1358008826561024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33784394665984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1324224374783488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1574240494898176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24323091166199 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1574237363112960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 450115056 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24325199959299 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245583057 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24328303428431 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245583057 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 _runtime‚11484.579281689 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1324336757513216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33787357454336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 450153456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884395062236204e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 450130318 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 450130318 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 245606189 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1547790463167311 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24369520462336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24366415055360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1358124057869312 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33787300356096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1547787334270211 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204547261 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1523462134310912 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1550004554459648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1550004580851200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚89627299 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1550004580851200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212004682 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 245606189 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204547261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122377915 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚11399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1358124143101952¾ðMm$ì+­N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 245606189 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1547790463167311 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZGDeviceStatsMonitor.on_train_batch_¹MÀ”%end/active_bytes.small_pool.allocated‚24369520462336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24366415055360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1358124057869312 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33787300356096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1547787334270211 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204547261 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1523462134310912 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1550004554459648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1550004580851200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚89627299 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1550004580851200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212004682 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 245606189 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204547261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122377915 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 trainer/global_step‚11399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1358124143101952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24325199959299 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245583057 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24328303428431 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245583057 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 _runtime‚11484.579281689 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1324336757513216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33787357454336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 450153456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884395062236204e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 450130318 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 450130318 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0©M ©J4í+/ _wandbruntime‚11484  _runtime‚11484r³QÊ_î+Ù /loss/FRt1/detection_loss‚1.5284751992852939 1loss/SRqqt1/detection_loss‚0.9192594497081111 %loss/total_loss‚42.50172654607799 % _timestamp‚1.7884395062253852e+09 _runtime‚11484.580365645 /loss/FRt2/assignment_loss‚3.585941676170595 0loss/FRt4/assignment_loss‚3.8070306594555197 1loss/FBt3/assignment_loss‚0.06516547666251427 1loss/SRqqt3/detection_loss‚1.4597254891333908 /loss/FBt2/detection_loss‚2.2373751362164813 /loss/FRt3/assignment_loss‚2.842504013807346 /loss/FRt3/detection_loss‚2.3660215456516314 /loss/FBt1/detection_loss‚0.8155055462740934 2loss/SRqqt2/assignment_loss‚1.2686532827577106 0loss/FRt1/assignment_loss‚1.4333052971670706 .loss/FRt2/detection_loss‚2.319825162810685 .loss/FRt4/detection_loss‚3.111465584773284 -loss/FBt3/detection_loss‚3.73162043094635 _step‚938 3loss/FBt1/assignment_loss‚0.0007258774684017383 2loss/FBt4/assignment_loss‚0.001270214709130904 /loss/SRqqt2/detection_loss‚2.54987672302458 /loss/FBt4/detection_loss‚1.9628504331295307 2loss/SRqqt1/assignment_loss‚0.9130327744792928 0loss/SRqqt4/detection_loss‚2.461271799036435 trainer/global_step‚11399 2loss/SRqqt4/assignment_loss‚1.6735853685864381 epoch‚9 2loss/SRqqt3/assignment_loss‚1.3555481383368244 1loss/FBt2/assignment_loss‚0.09169126648727446ºKñedï+Þ /loss/FRt3/assignment_loss‚2.842504013807346 /loss/FRt3/detection_loss‚2.3660215456516314 /loss/FBt1/detection_loss‚0.8155055462740934 2loss/SRqqt2/assignment_loss‚1.2686532827577106 0loss/FRt1/assignment_loss‚1.4333052971670706 .loss/FRt2/detection_loss‚2.319825162810685 .loss/FRt4/detection_loss‚3.111465584773284 -loss/FBt3/detection_loss‚3.73162043094635 _step‚938 3loss/FBt1/assignment_loss‚0.0007258774684017383 2loss/FBt4/assignment_loss‚0.001270214709130904 /loss/SRqqt2/detection_loss‚2.54987672302458 /loss/FBt4/detection_loss‚1.9628504331295307 2loss/SRqqt1/assignment_loss‚0.9130327744792928 0loss/SRqqt4/detection_loss‚2.461271799036435 trainer/global_step‚11399 2loss/SRqqt4/assignment_loss‚1.6735853685864381 epoch‚9 2loss/SRqqt3/assignment_loss‚1.3555481383368244 1loss/FBt2/assignment_loss‚0.09169126648727446 /loss/FRt1/detection_loss‚1.5284751992852939 1loss/SRqqt1/detection_loss‚0.9192594497081111 %loss/total_loss‚42.50172654607799 % _timestamp‚1.7884395062253852e+09 _runtime‚11484.580365645 /loss/FRt2/assignment_loss‚3.585941676170595 0loss/FRt4/assignment_loss‚3.8070306594555197 1loss/FBt3/assignment_loss‚0.06516547666251427 1loss/SRqqt3/detection_loss‚1.4597254891333908 /loss/FBt2/detection_loss‚2.2373751362164813ª¬ç “ð+‚(:ˆ ××åÔÃÜŠµ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Ö pñ+‚(:å ××åÔ¦ŠÀ· disk.drbd2080.out‚0 disk.sda1.in‚0% disk.nvme0n1.out‚83389.8720703125$ memory_percent‚99.17223453521729$ disk./.usageGB‚288.6209831237793 disk.drbd2265.in‚0 disk.dm-1.out‚ 428.84375( proc.memory.availableMB‚ 135.62109375 network.recv‚ 3.903854e+06* disk./.usagePercent‚19.369596346571775 disk.dm-0.in‚ 471.90234375 disk.sda.out‚ 428.84375 disk.drbd2265.out‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2891.1015625 disk.sda3.in‚0.203125 disk.sda3.out‚ 428.84375 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.645883560180664 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0" disk.dm-0.out‚83389.8720703125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 471.90234375 cpu‚25.280717545966002 network.sent‚ 2.932671e+07 disk.drbd1716.in‚0 disk.rbd1.out‚ 12.552734375 proc.cpu.threads‚17 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0' disk.nvme0c0n1.out‚83389.8720703125t|èÿ‘ò+‚(:† æ×åÔ—ê³µ gpu.0.gpu‚68.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚263.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.713 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v[Iòmó+‚(:â æ×åÔ¤ÏѶ" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚83429.9619140625 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda.out‚429.1875 disk.drbd2265.out‚0 network.sent‚ 2.943558e+07 disk.drbd2266.in‚0* proc.memory.percent‚17.647838592529297 disk.sda1.out‚0( proc.memory.availableMB‚ 135.30078125 cpu‚25.280421298776236 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0% disk./.usageGB‚288.63465881347656 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.dm-1.in‚0.203125 disk.rbd0.out‚0% disk.nvme0n1.out‚83429.9619140625 disk.dm-1.out‚429.1875! proc.memory.rssMB‚ 2891.421875 network.recv‚ 3.910121e+06" disk.dm-0.out‚83429.9619140625 disk.sda3.out‚429.1875 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 12.615234375$ memory_percent‚99.17418956756592* disk./.usagePercent‚19.370514133582027„é“ô+‚(:ˆ õ×åÔꬨ¶ gpu.0.gpu‚76.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚291.244' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žmDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1556447791172608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 246642925 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 _runtime‚11531.047715986 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1529802612798464 trainer/global_step‚11449 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246619819 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 246619819 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1529802587370496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24428544412407 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212878412 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24469932848128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24473038241792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205394183 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1363775489798656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1580917697629184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205394177 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24431647881331 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33930657054720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1580917697629184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246642925 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚940 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33930599943168 %àòUû+£P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 452037108 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1329844889855488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205394177 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1556447791172608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚89997552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122880315 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1556447764781056 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1580920829414400 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1363775575044608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884395526925187e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24473038241792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚89998097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246642925 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚940 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚33930599943168 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1556447764781056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24469932848128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1554234260679795 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1580920829414400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1556447791172608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 246642925 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 _runtime‚11531.047715986 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1529802612798464 trainer/global_step‚11449 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246619819 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 246619819 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 [LDeviceStatsMonitor.on_train_baqÏeÔtch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1529802587370496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24428544412407 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212878412 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24469932848128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24473038241792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205394183 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1363775489798656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1580917697629184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205394177 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24431647881331 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚33930657054720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1580917697629184 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212895724 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1529931984371712 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24475150568960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 246664041 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1329957300719616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122890565 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205411461 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33933568100864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33933511002624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 452052370 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 452075508 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1554362637577219 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 452052370 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1329957328854016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 452075508 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1581051304031232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24433756674639 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚90005704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚941 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24472045161984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 246640909 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1363890896954880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884395535924714e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205411461 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1554365766474319 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1556579285260800 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122890568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 246640909 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24430653205507 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1581054435829760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1556579258869248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 205411467 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1556579285260800 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1363890811722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205411467 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246664041 _runtime‚11531.948281506ÝDgÅ3'þ+­N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1581054435829760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1556579258869248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 205411467 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1556579285260800 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1363890811722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205411467 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246664041 _runtime‚11531.948281506 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212895724 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1529931984371712 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24475150568960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 246664041 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1329957300719616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122890565 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205411461 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚33933568100864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚33933511002624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 452052370 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 452075508 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1554362637577219 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 452052370 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1329957328854016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 452075508 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1581051304031232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24433756674639 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚90005704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 _step‚941 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24472045161984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 246640909 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1363890896954880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 % _timestamp‚1.7884395535924714e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205411461 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1554365766474319 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1556579285260800 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122890568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 246640909 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24430653205507 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180­Mf4ÿ+/ _wandbruntime‚11531  _runtime‚11531TLSôd€,Þ /loss/FBt2/detection_loss‚2.3731929153203963 trainer/global_step‚11449 0loss/FRt3/assignment_loss‚2.5610830728397813 4loss/FBt1/assignment_loss‚0.00026598321910423364 /loss/FBt1/detection_loss‚0.5836723025054916 /loss/FBt3/detection_loss‚2.7628943398594856 /loss/FBt4/detection_loss‚2.4562096860673694 _runtime‚11531.949504013 0loss/FRt1/assignment_loss‚2.1421931836069845 2loss/SRqqt1/assignment_loss‚0.8125815215957118 1loss/FBt2/assignment_loss‚0.13283074460923672 .loss/FRt3/detection_loss‚2.546795889388683 % _timestamp‚1.7884395535945318e+09 _step‚942 /loss/FRt4/assignment_loss‚3.252333205757719 1loss/FBt4/assignment_loss‚0.15853890383732505 1loss/SRqqt2/detection_loss‚2.2595977941527963 1loss/SRqqt3/detection_loss‚1.6254784395849262 1loss/SRqqt4/detection_loss‚2.2428913921117783 epoch‚9 2loss/SRqqt2/assignment_loss‚1.5085245054215193 2loss/SRqqt4/assignment_loss‚1.3811042005506655 -loss/FRt1/detection_loss‚1.68297853176387 1loss/SRqqt1/detection_loss‚1.0080388193018734 /loss/FRt2/assignment_loss‚2.572594199631665 /loss/FRt2/detection_loss‚2.5158635168462187 &loss/total_loss‚40.552357296180006 1loss/FBt3/assignment_loss‚0.07055120010954852 2loss/SRqqt3/assignment_loss‚1.1950285189129688 /loss/FRt4/detection_loss‚2.7071144291848848/LX…i,ã _runtime‚11531.949504013 0loss/FRt1/assignment_loss‚2.1421931836069845 2loss/SRqqt1/assignment_loss‚0.8125815215957118 1loss/FBt2/assignment_loss‚0.13283074460923672 .loss/FRt3/detection_loss‚2.546795889388683 % _timestamp‚1.7884395535945318e+09 _step‚942 /loss/FRt4/assignment_loss‚3.252333205757719 1loss/FBt4/assignment_loss‚0.15853890383732505 1loss/SRqqt2/detection_loss‚2.2595977941527963 1loss/SRqqt3/detection_loss‚1.6254784395849262 1loss/SRqqt4/detection_loss‚2.2428913921117783 epoch‚9 2loss/SRqqt2/assignment_loss‚1.5085245054215193 2loss/SRqqt4/assignment_loss‚1.3811042005506655 -loss/FRt1/detection_loss‚1.68297853176387 1loss/SRqqt1/detection_loss‚1.0080388193018734 /loss/FRt2/assignment_loss‚2.572594199631665 /loss/FRt2/detection_loss‚2.5158635168462187 &loss/total_loss‚40.552357296180006 1loss/FBt3/assignment_loss‚0.07055120010954852 2loss/SRqqt3/assignment_loss‚1.1950285189129688 /loss/FRt4/detection_loss‚2.7071144291848848 /loss/FBt2/detection_loss‚2.3731929153203963 trainer/global_step‚11449 0loss/FRt3/assignment_loss‚2.5610830728397813 4loss/FBt1/assignment_loss‚0.00026598321910423364 /loss/FBt1/detection_loss‚0.5836723025054916 /loss/FBt3/detection_loss‚2.7628943398594856 /loss/FBt4/detection_loss‚2.4562096860673694®ÏO `ž‚,‚(:“ „ØåԷ˲µ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àõ.Ixƒ,‚(:í „ØåÔáÙ„¸ disk.drbd2080.out‚0 network.recv‚ 3.918375e+06 disk.sda3.out‚ 430.26953125 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.rbd1.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚83534.044921875 disk.sda2.in‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.in‚0 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚83534.044921875# memory_percent‚98.7786054611206 disk.sda.out‚ 430.26953125( proc.memory.availableMB‚ 200.11328125 cpu‚25.279816941319392 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0$ disk./.usageGB‚288.6582717895508 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 430.26953125 disk.rbd1.out‚ 12.6474609375# proc.memory.rssMB‚ 2891.58203125* disk./.usagePercent‚19.372098820218973 disk.sda3.in‚0.203125* proc.memory.percent‚17.648816108703613 proc.cpu.threads‚17 network.sent‚ 2.9447193e+07! disk.dm-0.out‚83534.044921875 disk.drbd2080.in‚0è |Ñ“„,‚(:ˆ “ØåÔ½ûû´ gpu.0.gpu‚80.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚130.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á×Ur…,‚(:ç “ØåÔæߧ¸ disk.dm-0.in‚ 471.90234375 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2891.73828125* disk./.usagePercent‚19.373774901545644 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 471.90234375 network.recv‚ 3.924771e+06 disk.rbd1.out‚ 12.6474609375 disk.sda3.out‚430.9375& disk.nvme0n1.out‚83584.33251953125 disk.sda2.in‚0$ disk./.usageGB‚288.6832466125488 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125$ memory_percent‚98.78108501434326 cpu‚25.279517960389477 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚83584.33251953125 disk.dm-1.out‚430.9375 disk.drbd2266.out‚0) proc.memory.percent‚17.64976978302002 network.sent‚ 2.9556034e+07" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚83584.33251953125 disk.sda3.in‚0.203125 disk.sda.out‚430.9375( proc.memory.availableMB‚ 199.70703125®|ņ,‚(:’ ¢ØåÔ¥´µ gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚285.334' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.06679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P*ˆƒ‡,‚(:î ¢ØåԱó¶ disk.sda1.out‚0 disk.sda.in‚0.203125 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.rbd0.out‚0 disk.rbd1.out‚12.71044921875 network.recv‚ 3.928571e+06 disk.sda2.out‚0 disk.dm-1.out‚ 431.4140625( disk.nvme0c0n1.out‚83636.10595703125 disk.sda1.in‚0 disk.drbd2265.out‚0$ memory_percent‚98.78053665161133 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0# disk.dm-0.out‚83636.10595703125 disk.sda3.out‚ 431.4140625& proc.memory.availableMB‚ 199.796875" proc.memory.rssMB‚ 2891.8984375 proc.cpu.threads‚17 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 471.90234375& disk.nvme0n1.out‚83636.10595703125 disk.sda2.in‚0 disk.dm-0.in‚ 471.90234375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 431.4140625* proc.memþç öory.percent‚17.650747299194336 network.sent‚ 2.9561874e+07* disk./.usagePercent‚19.374809683999043$ disk./.usageGB‚288.6986656188965 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0 cpu‚25.27920801189368սͷ4ˆ,/ _wandbruntime‚11578  _runtime‚11578¾hËÚˉ,Å ' lr-AdamW/pg1‚1.4854770072623265e-06 ' lr-AdamW/pg2‚1.4854770072623265e-06 trainer/global_step‚11499 % _timestamp‚1.7884395999571192e+09 _runtime‚11578.312150965 _step‚943Æa¡ÐŠ,Ê ' lr-AdamW/pg1‚1.4854770072623265e-06 ' lr-AdamW/pg2‚1.4854770072623265e-06 trainer/global_step‚11499 % _timestamp‚1.7884395999571192e+09 _runtime‚11578.312150965 _step‚943¯!¯ÿ24‹,/ _wandbruntime‚11578  _runtime‚11578ÏR/$(Œ,žP JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 247700777 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247677671 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1563022469190656 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206258377 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1563022495582208 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1587598032145408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34076752653824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24578668348416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 206258383 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚90375192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34076695542272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247677671 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24575562954752 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1563022495582208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1587601163930624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1335465433061888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24533997658615 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247700777 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123392965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1536272437431296 _step‚944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1587598032145408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213767611 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1560806435089911 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 trainer/global_step‚11499 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536272462859264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1335465461196288 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1369542128604160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213768157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚90374649 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1369542213850112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚11578.313089797 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1587601163930624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24575562954752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206258377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1587601163930624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1335465433061888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24533997658615 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247700777 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123392965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1536272437431296 _step‚944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1587598032145408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213767611 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1560806435089911 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 trainer/global_step‚11499 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536272462859264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1335465461196288 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1369542128604160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213768157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚90374649 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1369542213850112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚11578.313089797 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1587601163930624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24575562954752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206258377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206275667 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24577675268608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213785878 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247698761 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1335577843926016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚11579.204664814 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1563153989670400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453974422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚90382660 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24577675268608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453997560 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247721893 % _timestamp‚1.7884396008488815e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1587731638547456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1587734770345984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1563153989670400 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1335577872060416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206275661 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34079613314560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚90382118 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1587734770345984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1560941069781327 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11499 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1536401859860480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206275661 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1563153963278848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34079668315648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213785333 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1369657457240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453997560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1563153963278848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24539209920847 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878õVåË,­N PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453974422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚90382660 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24577675268608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453997560 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247721893 % _timestamp‚1.7884396008488815e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1587731638547456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1587734770345984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1563153989670400 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1335577872060416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206275661 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34079613314560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚90382118 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1587734770345984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1560941069781327 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11499 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1536401859860480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206275661 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1563153963278848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34079668315648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213785333 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1369657457240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453997560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1563153963278848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24539209920847 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247721893 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247698761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206275667 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24577675268608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213785878 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247698761 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1335577843926016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _runtime‚11579.204664814 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1563153989670400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0±h˜g4‘,/ _wandbruntime‚11579  _runtime‚11579¢Ñ `’,Ú /loss/FRt3/assignment_loss‚2.175106092905387 1loss/FBt2/assignment_loss‚0.14579255159648408 /loss/FRt2/detection_loss‚2.2719959536114507 % _timestamp‚1.7884396008511493e+09 /loss/FRt2/assignment_loss‚3.191248795470676 -loss/FBt3/detection_loss‚2.68990296125412 &loss/total_loss‚40.755820810310325 0loss/FRt1/assignment_loss‚1.4558354571010128 3loss/FBt1/assignment_loss‚0.0003793241679353904 1loss/FBt3/assignment_loss‚0.08219628191242496 /loss/FRt3/detection_loss‚2.6071105343409076 1loss/SRqqt3/detection_loss‚1.9495550388736385 /loss/FBt2/detection_loss‚2.1439579367637633 trainer/global_step‚11499 /loss/FRt4/detection_loss‚2.7541656432480646 1loss/SRqqt2/detection_loss‚2.1914561034904585 1loss/SRqqt1/detection_loss‚0.9562613432459971 1loss/SRqqt4/detection_loss‚3.0331493452191354 /loss/FBt1/detection_loss‚0.6078550860619367 _step‚946 /loss/FRt4/assignment_loss‚3.916115968391813 2loss/SRqqt1/assignment_loss‚1.0795248617144193 2loss/SRqqt3/assignment_loss‚1.1064646876239705 1loss/FBt4/assignment_loss‚0.21181529210298322 /loss/FBt4/detection_loss‚1.9213097896426916 1loss/SRqqt2/assignment_loss‚1.217862522539993 2loss/SRqqt4/assignment_loss‚1.6847851322963834 .loss/FRt1/detection_loss‚1.361974106734675 epoch‚9 _runtime‚11579.20612502N:øQe“,ß 2loss/SRqqt4/assignment_loss‚1.6847851322963834 .loss/FRt1/detection_loss‚1.361974106734675 epoch‚9 _runtime‚11579.20612502 /loss/FRt3/assignment_loss‚2.175106092905387 1loss/FBt2/assignment_loss‚0.14579255159648408 /loss/FRt2/detection_loss‚2.2719959536114507 % _timestamp‚1.7884396008511493e+09 /loss/FRt2/assignment_loss‚3.191248795470676 -loss/FBt3/detection_loss‚2.68990296125412 &loss/total_loss‚40.755820810310325 0loss/FRt1/assignment_loss‚1.4558354571010128 3loss/FBt1/assignment_loss‚0.0003793241679353904 1loss/FBt3/assignment_loss‚0.08219628191242496 /loss/FRt3/detection_loss‚2.6071105343409076 1loss/SRqqt3/detection_loss‚1.9495550388736385 /loss/FBt2/detection_loss‚2.1439579367637633 trainer/global_step‚11499 /loss/FRt4/detection_loss‚2.7541656432480646 1loss/SRqqt2/detection_loss‚2.1914561034904585 1loss/SRqqt1/detection_loss‚0.9562613432459971 1loss/SRqqt4/detection_loss‚3.0331493452191354 /loss/FBt1/detection_loss‚0.6078550860619367 _step‚946 /loss/FRt4/assignment_loss‚3.916115968391813 2loss/SRqqt1/assignment_loss‚1.0795248617144193 2loss/SRqqt3/assignment_loss‚1.1064646876239705 1loss/FBt4/assignment_loss‚0.21181529210298322 /loss/FBt4/detection_loss‚1.9213097896426916 1loss/SRqqt2/assignment_loss‚1.217862522539993²0±”,‚(:’ ±ØåÔôŠ¦µ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.857' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.37140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÈLÖx•,‚(:í ±ØåÔûÖ· proc.cpu.threads‚17 network.recv‚ 3.93336e+06 disk.sda.in‚0.203125 disk.sda.out‚ 431.94921875 disk.nvme0n1.in‚ 471.90234375" proc.memory.rssMB‚ 2891.8984375 disk.drbd1716.out‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.out‚12.71044921875 cpu‚25.278908975803997( disk./.usagePercent‚19.3771488248394 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0% disk.nvme0n1.out‚83693.5439453125 disk.dm-1.out‚ 431.94921875' disk.nvme0c0n1.out‚83693.5439453125$ memory_percent‚98.78220558166504 disk.dm-0.in‚ 471.90234375 disk.sda3.in‚0.203125* proc.memory.percent‚17.650747299194336 network.sent‚ 2.9568318e+07 disk.drbd2265.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 199.5234375 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0" disk.dm-0.out‚83693.5439453125 disk.sda1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 471.90234375$ disk./.usageGB‚288.7335205078125 disk.sda3.out‚ 431.94921875 disk.sda2.out‚0 disk.drbd2266.out‚0 5âk“–,‚(:ˆ ÀØåÔóÍѵ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.4252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÈHfv—,‚(:ë ÀØåԯබ$ memory_percent‚98.55356216430664 proc.cpu.threads‚17 disk.dm-0.in‚ 471.90234375 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.650747299194336 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 236.984375" proc.memory.rssMB‚ 2891.8984375 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 432.35546875 disk.sda1.out‚0 network.recv‚ 3.938353e+06% disk./.usageGB‚288.74546813964844 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0% disk.nvme0n1.out‚83735.0361328125) disk./.usagePercent‚19.37795064043664 disk.rbd1.out‚ 12.7734375 disk.rbd0.out‚0 network.sent‚ 2.9676482e+07 disk.drbd2265.out‚0 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375' disk.nvme0c0n1.out‚83735.0361328125" disk.dm-0.out‚83735.0361328125 disk.drbd2265.in‚0 cpu‚25.278637693877755 disk.sda.in‚0.203125 disk.sda.out‚ 432.35546875 disk.rbd1.in‚0 disk.sda3.out‚ 432.35546875 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 471.90234375Ú€˜,‚(:’ ÏØåÔØ»“µ gpu.0.gpu‚85.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚259.417' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.88339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú%|Hl™,‚(:á ÏØåÔïœÒ¶ disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0$ memory_percent‚98.55546951293945 proc.cpu.threads‚17 disk.dm-0.out‚83787.41015625 disk.drbd1716.in‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375# disk.nvme0n1.out‚83787.41015625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 236.671875 disk.sda3.out‚ 433.0546875 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚83787.41015625" disk.nvme0c0n1.in‚ 471.90234375 disk.dm-0.in‚ 471.90234375 network.recv‚ 3.942848e+06 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 433.0546875" proc.memory.rssMB‚ 2892.2109375) proc.memory.percent‚17.65265464782715 network.sent‚ 2.9682518e+07* disk./.usagePercent‚19.378600387558542$ disk./.usageGB‚288.7551498413086 disk.sda.out‚ 433.0546875 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 12.7734375 disk.sda3.in‚0.203125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.203125 cpu‚25.278341967596027Ý«þJ’š,‚(:‡ ÞØåÔþ‹“µ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚294.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.897 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úLèàz›,‚(:ï ÞØåÔ«€· disk.sda2.in‚0 disk.dm-0.in‚ 471.90234375 disk.sda3.out‚ 433.66796875 disk.drbd2080.in‚0 disk.sda.out‚ 433.66796875 disk.drbd2265.out‚0& disk.nvme0n1.out‚83845.91259765625 cpu‚25.27804634113931 disk.rbd0.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.drbd1716.in‚0 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.drbd2080.out‚0( proc.memory.availableMB‚ 236.66796875 network.sent‚ 2.9688385e+07 network.recv‚ 3.947148e+06 disk.nvme0n1.in‚ 471.90234375 disk.dm-1.out‚ 433.66796875 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.203125 disk.sda2.out‚0( disk.nvme0c0n1.out‚83845.91259765625 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 12.7734375# disk.dm-0.out‚83845.91259765625$ memory_percent‚98.55549335479736% disk./.usageGB‚288.77368545532227 disk.rbd1.in‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2892.2109375) proc.memory.percent‚17.65265464782715* disk./.usagePercent‚19.379844328167415 disk.rbd0.out‚0ï¨4œ,/ _wandbruntime‚11625  _runtime‚11625„ÜòÉ,à ' lr-AdamW/pg1‚1.2452493878169043e-06 ' lr-AdamW/pg2‚1.2452493878169043e-06 trainer/global_step‚11549 # _timestamp‚1.78843964699991e+09 _runtime‚11625.355001574 _step‚947.ÝüïΞ,È ' lr-AdamW/pg2‚1.2452493878169043e-06 trainer/global_step‚11549 # _timestamp‚1.78843964699991e+09 _runtime‚11625.355001574 _step‚947 ' lr-AdamW/pg1‚1.2452493878169043e-06³/-4Ÿ,/ _wandbruntime‚11625  _runtime‚11625DŠ}–#( ,P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207122577 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207122577 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1375309063330304 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1341086004402688 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248758629 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214661456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1567384867293811 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1341085976268288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24639450904823 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34223058927616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248758629 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1569597173600256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207122583 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214660885 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123905615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1567381738396919 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚11625.355989649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1569597173600256 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1594278366661632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚11549 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 $ _timestamp‚1.788439647000751e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚90755273 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1594281498446848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1594278366661632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34222999718912 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24681193061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24684298455040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123905612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1542742312920064 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1542742287492096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 $ _timestamp‚1.788439647000751e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚90755273 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1594281498446848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1594278366661632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34222999718912 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 H?Device˜HônStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24681193061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24684298455040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123905612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1542742312920064 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1542742287492096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207122577 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1375309063330304 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1341086004402688 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248758629 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214661456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1567384867293811 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1341085976268288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚87343104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24639450904823 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34223058927616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248758629 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1569597173600256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207122583 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214660885 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123905615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1567381738396919 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚11625.355989649´ÏyAB4¢,/ _wandbruntime‚11626  _runtime‚11626œ?–.'£,¨N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207139861 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24683305375232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1594415104862208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1569728667688448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214679142 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1567513244191235 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚11549 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34225909577216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455919612 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24641559697923 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248756613 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123915868 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248756613 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207139861 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455896474 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1569728694080000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207139867 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1542871684493312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24644663167055 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24686410782208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1567516373088335 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248779745 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 455896474 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207139867 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1341198387132416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34225966675456 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1542871709921280 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214678589 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 455919612 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1569728694080000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248779745 % _timestamp‚1.7884396478941145e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0jl.3'¤,­N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207139861 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455896474 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1569728694080000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207139867 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1542871684493312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24644663167055 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24686410782208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1567516373088335 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248779745 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 455896474 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207139867 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1341198387132416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34225966675456 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1542871709921280 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214678589 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 455919612 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1569728694080000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248779745 % _timestamp‚1.7884396478941145e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207139861 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24683305375232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1594415104862208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1569728667688448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214679142 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1567513244191235 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 trainer/global_step‚11549 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34225909577216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455919612 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24641559697923 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248756613 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123915868 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248756613µõ©jY4¥,/ _wandbruntime‚11626  _runtime‚11626$Ì(a¦,Û _step‚950 1loss/FBt2/assignment_loss‚0.09110866144496518 /loss/FRt3/detection_loss‚2.4809396471828222 .loss/FRt4/detection_loss‚2.767718334992727 1loss/SRqqt2/detection_loss‚2.3562065148726106 /loss/FBt1/detection_loss‚0.6817516142671759 .loss/FBt3/detection_loss‚2.691964098385402 %loss/total_loss‚39.83681818436617 .loss/FRt2/detection_loss‚2.240379836629442 2loss/SRqqt4/assignment_loss‚1.4664183010657628 1loss/FBt3/assignment_loss‚0.29728558607996547 _runtime‚11626.251284516 /loss/FBt2/detection_loss‚1.8551226074878986 .loss/FBt4/detection_loss‚1.865229614756324 epoch‚9 1loss/SRqqt3/detection_loss‚1.7473639201210893 1loss/SRqqt4/detection_loss‚2.2896287206447488 2loss/SRqqt2/assignment_loss‚1.5459131901152432 2loss/SRqqt3/assignment_loss‚1.2527629149996715 3loss/FBt1/assignment_loss‚0.0018751168809161682 /loss/FRt1/detection_loss‚1.4673521793846573 % _timestamp‚1.7884396478963056e+09 /loss/FRt2/assignment_loss‚2.929606060771381 /loss/FRt3/assignment_loss‚2.862935072183609 0loss/FRt4/assignment_loss‚3.8323127737751714 2loss/SRqqt1/assignment_loss‚0.8498413708429177 1loss/FBt4/assignment_loss‚0.09425747937480237 0loss/FRt1/assignment_loss‚1.3210550163473402 1loss/SRqqt1/detection_loss‚0.8477895517595285 trainer/global_step‚11549­àÇ#f§,à 1loss/SRqqt3/detection_loss‚1.7473639201210893 1loss/SRqqt4/detection_loss‚2.2896287206447488 2loss/SRqqt2/assignment_loss‚1.5459131901152432 2loss/SRqqt3/assignment_loss‚1.2527629149996715 3loss/FBt1/assignment_loss‚0.0018751168809161682 /loss/FRt1/detection_loss‚1.4673521793846573 % _timestamp‚1.7884396478963056e+09 /loss/FRt2/assignment_loss‚2.929606060771381 /loss/FRt3/assignment_loss‚2.862935072183609 0loss/FRt4/assignment_loss‚3.8323127737751714 2loss/SRqqt1/assignment_loss‚0.8498413708429177 1loss/FBt4/assignment_loss‚0.09425747937480237 0loss/FRt1/assignment_loss‚1.3210550163473402 1loss/SRqqt1/detection_loss‚0.8477895517595285 trainer/global_step‚11549 _step‚950 1loss/FBt2/assignment_loss‚0.09110866144496518 /loss/FRt3/detection_loss‚2.4809396471828222 .loss/FRt4/detection_loss‚2.767718334992727 1loss/SRqqt2/detection_loss‚2.3562065148726106 /loss/FBt1/detection_loss‚0.6817516142671759 .loss/FBt3/detection_loss‚2.691964098385402 %loss/total_loss‚39.83681818436617 .loss/FRt2/detection_loss‚2.240379836629442 2loss/SRqqt4/assignment_loss‚1.4664183010657628 1loss/FBt3/assignment_loss‚0.29728558607996547 _runtime‚11626.251284516 /loss/FBt2/detection_loss‚1.8551226074878986 .loss/FBt4/detection_loss‚1.865229614756324 epoch‚9¶o«ä¨,‚(:’ íØåÔ՜ݵ gpu.0.gpu‚96.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚148.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.766000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÎC8t©,‚(:é íØåÔùÖ·¸ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.rbd1.out‚ 12.7734375 disk.drbd2265.in‚0* proc.memory.percent‚17.653632164001465 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚83903.97314453125 disk.sda3.out‚ 434.390625 disk.drbd1716.out‚0 disk.sda.in‚0.203125# proc.memory.rssMB‚ 2892.37109375& disk.nvme0n1.out‚83903.97314453125 disk.rbd1.in‚0( disk.nvme0c0n1.out‚83903.97314453125 disk.sda1.out‚0' proc.memory.availableMB‚ 236.2265625 disk.drbd1716.in‚0 network.sent‚ 2.969469e+07$ disk./.usageGB‚288.7852096557617 disk.nvme0n1.in‚ 471.90234375" disk.nvme0c0n1.in‚ 471.90234375 disk.sda3.in‚0.203125 disk.sda.out‚ 434.390625 cpu‚25.277768112280565* disk./.usagePercent‚19.380617726928357 disk.drbd2266.out‚0 disk.drbd2265.out‚0$ memory_percent‚98.55818748474121 disk.dm-1.out‚ 434.390625 disk.drbd2266.in‚0 proc.cpu.threads‚17 network.recv‚ 3.951666e+06uÓÏ,‚(:“ üØåÔ©ù¶ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚45.0 gpu.0.powerWatts‚154.153' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.830599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ìIXu«,‚(:ê üØåÔõÓï¹ disk.drbd1716.in‚0 disk.drbd2265.in‚0# memory_percent‚98.5576868057251( proc.memory.availableMB‚ 236.30859375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚288.7951889038086 disk.sda.in‚0.203125& disk.nvme0n1.out‚83945.35498046875 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 471.90234375# disk.dm-0.out‚83945.35498046875) proc.memory.percent‚17.65460968017578 cpu‚25.27746910347756 disk.sda3.out‚ 434.91796875 disk.sda1.out‚0 disk.sda.out‚ 434.91796875 disk.sda2.in‚0 disk.dm-1.out‚ 434.91796875( disk.nvme0c0n1.out‚83945.35498046875 proc.cpu.threads‚17 network.sent‚ 2.9803041e+07 disk.sda3.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.rbd1.out‚ 12.7734375 network.recv‚ 3.957094e+06 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚19.38128744263793 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2892.53125 disk.dm-0.in‚ 471.90234375ØÌ.Á¬,‚(:’ ‹ÙåÔ˺¶ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚294.78' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.955999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß´)P?­,‚(:Ø ‹ÙåÔÓò„· proc.cpu.threads‚17 network.sU¤‡‘$ent‚ 2.9809116e+07 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚435.5# disk.dm-0.out‚83996.52490234375$ memory_percent‚98.55921268463135 disk.rbd0.out‚0& disk.nvme0n1.out‚83996.52490234375 disk.sda2.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 236.05859375 disk.sda.out‚435.5 disk.drbd2265.in‚0) proc.memory.percent‚17.65460968017578 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda3.in‚0.203125 disk.sda3.out‚435.5 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 12.7734375 cpu‚25.27719015089856* disk./.usagePercent‚19.382046760984483 disk.rbd0.in‚0 disk.dm-0.in‚ 471.90234375 disk.drbd1716.out‚0 network.recv‚ 3.961681e+06% disk./.usageGB‚288.80650329589844 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚83996.52490234375 proc.memory.rssMB‚ 2892.53125ûûp4®,/ _wandbruntime‚11672  _runtime‚11672Š5:ȯ, & lr-AdamW/pg1‚1.026078345408791e-06 & lr-AdamW/pg2‚1.026078345408791e-06 trainer/global_step‚11599 % _timestamp‚1.7884396942471101e+09 _runtime‚11672.60216146 _step‚951¹µë:Í°,Ç trainer/global_step‚11599 % _timestamp‚1.7884396942471101e+09 _runtime‚11672.60216146 _step‚951 & lr-AdamW/pg1‚1.026078345408791e-06 & lr-AdamW/pg2‚1.026078345408791e-06·ç©íñ4±,/ _wandbruntime‚11672  _runtime‚11672lùm$(²,žP BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 457780152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24789928561664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚952 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1576171904401408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24744904151031 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24789928561664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24786823168000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚91131314 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207986777 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1576171904401408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124418262 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24786823168000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1549212162980864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1600961832963072 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1346706547609088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 249793375 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1600961832963072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215550126 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124418265 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1573957041703927 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1600958701177856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚91131861 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249816481 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34368978255360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 249793375 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1600961832963072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215550126 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124418265 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1573957041703927 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1600958701177856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚91131861 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249816481 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34368978255360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 457780152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24789928561664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 _step‚952 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1576171904401408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24744904151031 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24789928561664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24786823168000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚91131314 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207986777 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1576171904401408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124418262 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180¸²+M4´,/ _wandbruntime‚11673  _runtime‚11673X~Ó.'µ,¨N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 457818526 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208004061 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚24750116413263 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 457841664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚953 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 457841664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24788935481856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1576303372098048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208004067 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11599 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208004061 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249814465 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1346818958473216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215567512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚514 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1549341559982080 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24747012944131 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1601095439378432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208004067 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 _runtime‚11673.496842859 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1574088547498243 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249814465 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚91138997 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24788935481856 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124428515 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24792040888832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34371912415232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884396951411822e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1549341534554112 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34371855316992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1601095439378432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚91139511 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1346818930338816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1576303372098048 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1549341534554112 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34371855316992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1601095439378432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚91139511 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1346818930338816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1576303372098048 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 457841664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚953 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 457841664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24788935481856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1576303372098048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208004067 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11599 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208004061 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249814465 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1346818958473216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215567512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚514¹F9%È4·,/ _wandbruntime‚11673  _runtime‚11673ëà _¸,Ù 0loss/FRt1/assignment_loss‚1.2483406809911335 /loss/FRt2/assignment_loss‚2.746985038644389 0loss/FRt3/assignment_loss‚2.6165727782756725 2loss/SRqqt3/assignment_loss‚1.4683482640078052 0loss/FBt2/assignment_loss‚0.4723500131199566 .loss/FBt3/detection_loss‚1.925373951594035 trainer/global_step‚11599 1loss/SRqqt1/detection_loss‚1.0350749616551658 1loss/SRqqt2/detection_loss‚2.4029227952872003 /loss/FBt4/detection_loss‚2.1157930703724133 1loss/FBt3/assignment_loss‚0.31795875752059843 epoch‚9 % _timestamp‚1.7884396951429498e+09 0loss/FRt4/assignment_loss‚2.8771082735061646 2loss/FBt1/assignment_loss‚0.004833030552904357 /loss/FRt4/detection_loss‚3.0040383386611937 1loss/FBt4/assignment_loss‚0.14273474468192196 .loss/FRt1/detection_loss‚1.354238667282952 1loss/SRqqt3/detection_loss‚1.5694925867553269 0loss/SRqqt4/detection_loss‚2.321401401928493 /loss/FBt1/detection_loss‚0.6339117180489536 $loss/total_loss‚39.6845351684711 _runtime‚11673.497918529 _step‚954 2loss/SRqqt1/assignment_loss‚1.0245695024871342 2loss/SRqqt2/assignment_loss‚1.7162117178418808 2loss/SRqqt4/assignment_loss‚1.3968557088502815 /loss/FRt2/detection_loss‚2.7063221962828385 .loss/FRt3/detection_loss‚2.197089188910545 /loss/FBt2/detection_loss‚2.3860077812121463uMŠzd¹,Þ 0loss/FBt2/assignment_loss‚0.4723500131199566 .loss/FBt3/detection_loss‚1.925373951594035 trainer/global_step‚11599 1loss/SRqqt1/detection_loss‚1.0350749616551658 1loss/SRqqt2/detection_loss‚2.4029227952872003 /loss/FBt4/detection_loss‚2.1157930703724133 1loss/FBt3/assignment_loss‚0.31795875752059843 epoch‚9 % _timestamp‚1.7884396951429498e+09 0loss/FRt4/assignment_loss‚2.8771082735061646 2loss/FBt1/assignment_loss‚0.004833030552904357 /loss/FRt4/detection_loss‚3.0040383386611937 1loss/FBt4/assignment_loss‚0.14273474468192196 .loss/FRt1/detection_loss‚1.354238667282952 1loss/SRqqt3/detection_loss‚1.5694925867553269 0loss/SRqqt4/detection_loss‚2.321401401928493 /loss/FBt1/detection_loss‚0.6339117180489536 $loss/total_loss‚39.6845351684711 _runtime‚11673.497918529 _step‚954 2loss/SRqqt1/assignment_loss‚1.0245695024871342 2loss/SRqqt2/assignment_loss‚1.7162117178418808 2loss/SRqqt4/assignment_loss‚1.3968557088502815 /loss/FRt2/detection_loss‚2.7063221962828385 .loss/FRt3/detection_loss‚2.197089188910545 /loss/FBt2/detection_loss‚2.3860077812121463 0loss/FRt1/assignment_loss‚1.2483406809911335 /loss/FRt2/assignment_loss‚2.746985038644389 0loss/FRt3/assignment_loss‚2.6165727782756725 2loss/SRqqt3/assignment_loss‚1.4683482640078052ºKÈ‘º,‚(:† šÙåÔÖà´µ gpu.0.gpu‚44.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚129.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:[t»,‚(:é šÙåÔŽ£ï¶ disk.drbd2080.out‚0& disk.nvme0n1.out‚84046.50537109375 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 435.9765625( disk.nvme0c0n1.out‚84046.50537109375 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2892.53125* disk./.usagePercent‚19.378309050984576 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd2080.in‚0$ memory_percent‚98.55806827545166) proc.memory.percent‚17.65460968017578 cpu‚25.276878554299458 proc.cpu.threads‚17 network.sent‚ 2.9815153e+07 disk.dm-0.in‚ 471.90234375# disk.dm-0.out‚84046.50537109375 disk.sda.in‚0.203125$ disk./.usageGB‚288.7508087158203 disk.nvme0n1.in‚ 471.90234375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda.out‚ 435.9765625 disk.rbd0.in‚0( proc.memory.availableMB‚ 236.24609375 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 13.328125 disk.sda3.out‚ 435.9765625 disk.drbd2265.out‚0 network.recv‚ 3.966163e+06 disk.sda2.in‚0+É䯓¼,‚(:ˆ ©ÙåÔùÊŽµ gpu.0.gpu‚81.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚270.003' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gyŠîq½,‚(:æ ©ÙåÔóìì¶ disk.dm-0.in‚ 471.90234375 disk.drbd2080.in‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 471.90234375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.65460968017578% disk./.usageGB‚288.78843688964844 disk.rbd1.out‚ 13.328125# disk.dm-0.out‚84111.82568359375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 471.90234375 disk.drbd2266.out‚0" proc.memory.availableMB‚235.75 network.recv‚ 3.972771e+06 disk.sda.in‚0.203125 cpu‚25.27660973026517 network.sent‚ 2.9924308e+07 disk.sda1.out‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚84111.82568359375 disk.dm-1.out‚ 436.69140625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.out‚ 436.69140625 disk.sda2.out‚0* disk./.usagePercent‚19.380834309302323( disk.nvme0c0n1.out‚84111.82568359375 disk.drbd2080.out‚0 disk.sda.out‚ 436.69140625 disk.drbd2265.in‚0$ memory_percent‚98.56109619140625 proc.memory.rssMB‚ 2892.53125 disk.sda3.in‚0.203125 disk.dm-1.in‚0.203125fÊ~±“¾,‚(:ˆ ¸ÙåԷ䊵 gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.912' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i‘zf¿,‚(:Û ¸ÙåÔ¤ëä· disk.drbd2265.out‚0 disk.nvme0n1.in‚472.125& disk.nvme0n1.out‚84177.28076171875 disk.sda3.in‚0.203125) disk./.usagePercent‚19.38181456215093 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0# disk.dm-0.out‚84177.28076171875 disk.drbd1716.in‚0 network.sent‚ 2.9929583e+07 disk.dm-1.in‚0.203125 disk.nvme0c0n1.in‚472.125 disk.sda3.out‚ 437.33984375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.out‚ 437.33984375 disk.rbd0.out‚0( proc.memory.availableMB‚ 313.08203125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚84177.28076171875 proc.memory.rssMB‚ 2892.6875 disk.dm-1.out‚ 437.33984375 disk.rbd1.in‚0 disk.dm-0.in‚472.125 disk.sda1.in‚0$ memory_percent‚98.08909893035889* proc.memory.percent‚17.655563354492188 cpu‚25.276316047073543 network.recv‚ 3.976044e+06 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚13.40625 proc.cpu.threads‚17$ disk./.usageGB‚288.8030433654785¢¼šô4À,/ _wandbruntime‚11720  _runtime‚11720ì÷ÇÉÁ,à % _timestamp‚1.7884397416953568e+09 _runtime‚11720.050542951 _step‚955 & lr-AdamW/pg1‚8.280095549546942e-07 & lr-AdamW/pg2‚8.280095549546942e-07 trainer/global_step‚11649‹|ÎÂ,È & lr-AdamW/pg1‚8.280095549546942e-07 & lr-AdamW/pg2‚8.280095549546942e-07 trainer/global_step‚11649 % _timestamp‚1.7884397416953568e+09 _runtime‚11720.050542951 _step‚955»V®¨q4Ã,/ _wandbruntime‚11720  _runtime‚11720¹¼â:$(Ä,žP [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24892453274624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216440457 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1607642167479296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚24895558668288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1555681987613696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34515118723584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250874333 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250851227 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚91509545 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1386842264553472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 % _timestamp‚1.7884397416960819e+09 _step‚956 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚55014400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1607639035694080 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1582746582419456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1582746582419456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124930915 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1580532345010935 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124930912 _runtime‚11720.051272377 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 459725316 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208850977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1352327090815488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208850983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚24895558668288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1580535473907827 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24892453274624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 208850983 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1555682013041664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1582746608811008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1607642167479296 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1607639035694080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚91510082 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24853460866163 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216440997 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24850357397239 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208850977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1582746608811008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1386842181404672 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1352327062681088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250874333 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1607639035694080 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1582746582419456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1582746582419456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124930915 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1580532345010935 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124930912 _runtime‚11720.051272377 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 459725316 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208850977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1352327090815488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208850983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚2489555866àÝíþ8288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1580535473907827 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚24892453274624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 208850983 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1555682013041664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1582746608811008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1607642167479296 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1607639035694080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚91510082 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚24853460866163 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216440997 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24850357397239 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208850977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1582746608811008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1386842181404672 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1352327062681088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250874333 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984¼ Ÿ%4Æ,/ _wandbruntime‚11720  _runtime‚11720>ûÞ°.'Ç,¨N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250872317 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚24894565588480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250872317 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚91517452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 459740578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124941165 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208868267 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34518112833536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24852466190339 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1582878076507648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 _step‚957 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1580663850805251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 % _timestamp‚1.7884397425874755e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208868261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1386957529280512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24894565588480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 459763716 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 459740578 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250895449 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1582878102899200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216458107 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1607772642096128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24897670995456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1607775773894656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1582878076507648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124941168 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1352439501679616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1555811410042880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34518055735296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1582878102899200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24897670995456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124941165 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208868267 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34518112833536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24852466190339 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1582878076507648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 _step‚957 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1580663850805251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 % _timestamp‚1.7884397425874755e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208868261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1386957529280512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚24894565588480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 459763716 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 459740578 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250895449 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1582878102899200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216458107 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1607772642096128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚24897670995456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1607775773894656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1582878076507648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124941168 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1352439501679616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1555811410042880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34518055735296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1582878102899200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚24897670995456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24998083381248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209715177 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1614322501995520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1562151863102464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24955810643447 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1589321313220608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1587110777214835 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1392609061512192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209715183 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1587107648317943 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1357947605887488 _step‚960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1614322501995520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209715183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34661427490304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1614319370210304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1614319370210304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 217331225 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125443562 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34661372475904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1357947634021888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 251909079 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚91887660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217330684 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1392608978363392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1614319370210304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 217331225 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125443562 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34661372475904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1357947634021888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 251909079 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚91887660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217330684 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1392608978363392 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚24998083381248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209715177 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1614322501995520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1562151863102464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚24955810643447 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1589321313220608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1587110777214835 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1392609061512192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209715183 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1587107648317943 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1357947605887488 _step‚960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1614322501995520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209715183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34661427490304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1614319370210304 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25003301102080 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1614456108410880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1392724299837952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25000195695104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1587239154112259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217348400 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125453818 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34664283086336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚91894585 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1614452976612352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 461662630 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209732467 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217348912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125453815 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24957919436547 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 461685768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1587242283009359 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 461662630 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34664338087424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1562281260103680 % _timestamp‚1.7884397898570726e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 251930169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25000195695104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1358060016751616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209732461 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1589452780917248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25003301102080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1589452807308800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209732461 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25003301102080 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1614456108410880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1392724299837952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25000195695104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1587239154112259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217348400 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125453818 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚34664283086336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚91894585 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1614452976612352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 461662630 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209732467 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217348912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125453815 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚83135488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚55001088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚24957919436547 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 461685768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1587242283009359 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 461662630 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34664338087424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1562281260103680 % _timestamp‚1.7884397898570726e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 251930169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25000195695104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1358060016751616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209732461Á0ûú4Û,/ _wandbruntime‚11768  _runtime‚11768¹ª‰P^Ü,Ø /loss/FBt4/detection_loss‚2.5105647146701813 epoch‚9 0loss/FRt4/assignment_loss‚2.5512816508611045 2loss/SRqqt4/assignment_loss‚1.2982325304684967 /loss/FRt1/detection_loss‚1.4371966288333209 /loss/FRt4/detection_loss‚2.9445782436264887 _step‚962 /loss/FRt2/assignment_loss‚3.284791597613582 /loss/FBt2/detection_loss‚2.1889343857765198 $ _timestamp‚1.788439789858941e+09 0loss/FRt1/assignment_loss‚1.3415092943419873 1loss/SRqqt2/assignment_loss‚1.335986649561109 /loss/FRt2/detection_loss‚2.7233736007301896 1loss/SRqqt1/detection_loss‚0.7600026952123708 0loss/SRqqt2/detection_loss‚2.363845649993781 0loss/SRqqt3/detection_loss‚2.363672272861004 2loss/SRqqt3/assignment_loss‚1.2536926480941475 /loss/FBt1/detection_loss‚0.5861665295240911 2loss/SRqqt1/assignment_loss‚0.8499399380590976 1loss/FBt2/assignment_loss‚0.12393914222484455 1loss/FBt3/assignment_loss‚0.04497025876272685 .loss/FRt3/detection_loss‚3.628556278489885 0loss/FRt3/assignment_loss‚2.4167863386017934 _runtime‚11768.213908383 4loss/FBt1/assignment_loss‚0.00033650834777139497 -loss/FBt3/detection_loss‚1.91401321547372 %loss/total_loss‚40.39008379354456 trainer/global_step‚11699 0loss/FBt4/assignment_loss‚0.2621352830901742 0loss/SRqqt4/detection_loss‚2.205577738326171™¥c~cÝ,Ý epoch‚9 0loss/FRt4/assignment_loss‚2.5512816508611045 2loss/SRqqt4/assignment_loss‚1.2982325304684967 /loss/FRt1/detection_loss‚1.4371966288333209 /loss/FRt4/detection_loss‚2.9445782436264887 _step‚962 /loss/FRt2/assignment_loss‚3.284791597613582 /loss/FBt2/detection_loss‚2.1889343857765198 $ _timestamp‚1.788439789858941e+09 0loss/FRt1/assignment_loss‚1.3415092943419873 1loss/SRqqt2/assignment_loss‚1.335986649561109 /loss/FRt2/detection_loss‚2.7233736007301896 1loss/SRqqt1/detection_loss‚0.7600026952123708 0loss/SRqqt2/detection_loss‚2.363845649993781 0loss/SRqqt3/detection_loss‚2.363672272861004 2loss/SRqqt3/assignment_loss‚1.2536926480941475 /loss/FBt1/detection_loss‚0.5861665295240911 2loss/SRqqt1/assignment_loss‚0.8499399380590976 1loss/FBt2/assignment_loss‚0.12393914222484455 1loss/FBt3/assignment_loss‚0.04497025876272685 .loss/FRt3/detection_loss‚3.628556278489885 0loss/FRt3/assignment_loss‚2.4167863386017934 _runtime‚11768.213908383 4loss/FBt1/assignment_loss‚0.00033650834777139497 -loss/FBt3/detection_loss‚1.91401321547372 %loss/total_loss‚40.39008379354456 trainer/global_step‚11699 0loss/FBt4/assignment_loss‚0.2621352830901742 0loss/SRqqt4/detection_loss‚2.205577738326171 /loss/FBt4/detection_loss‚2.5105647146701813ÂMzGpÞ,‚(:’ ôÙåԶ gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚303.642' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.72840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©6~Íbß,‚(:× ôÙåÔ‡–Ù¶ disk.drbd2266.in‚0 disk.drbd2080.out‚0 cpu‚25.275149011497405 network.sent‚ 3.0055946e+07 disk.rbd0.in‚0# proc.memory.rssMB‚ 2892.84765625 proc.cpu.threads‚17 network.recv‚ 3.994847e+06% disk.nvme0n1.out‚84340.8232421875 disk.drbd2266.out‚0 disk.drbd2265.in‚0$ memory_percent‚98.09064865112305& proc.memory.availableMB‚ 312.828125* disk./.usagePercent‚19.385506446802207 disk.sda3.out‚ 439.84375 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚472.125 disk.drbd2080.in‚0 disk.sda.in‚0.203125* proc.memory.percent‚17.656540870666504 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 14.1650390625 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 439.84375' disk.nvme0c0n1.out‚84340.8232421875" disk.dm-0.out‚84340.8232421875 disk.sda.out‚ 439.84375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚472.125 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.out‚0$ disk./.usageGB‚288.8580551147461 disk.nvme0n1.in‚472.125šŽíþ“à,‚(:ˆ ƒÚåÔ˜­í´ gpu.0.gpu‚71.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚298.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.7116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 &ª­ná,‚(:ã ƒÚåÔéÁÇ·# proc.memory.rssMB‚ 2892.84765625 network.recv‚ 4.000613e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 14.2587890625$ memory_percent‚98.09062480926514* proc.memory.percent‚17.656540870666504 disk.sda3.out‚ 440.6640625 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 3.0164016e+07% disk./.usageGB‚288.88055419921875 disk.nvme0c0n1.in‚472.125( disk.nvme0c0n1.out‚84398.53857421875 disk.sda3.in‚0.203125 disk.drbd1716.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚84398.53857421875 disk.dm-1.out‚ 440.6640625 disk.drbd2266.in‚0 cpu‚25.274859401789133* disk./.usagePercent‚19.387016379239157 disk.drbd2266.out‚0# disk.dm-0.out‚84398.53857421875 disk.sda.out‚ 440.6640625 disk.nvme0n1.in‚472.125( proc.memory.availableMB‚ 312.83203125 proc.cpu.threads‚17 disk.dm-0.in‚472.125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd1.in‚0§?OR‘â,‚(:† ’ÚåÔûíɵ gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.78' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ygbÄlã,‚(:á ’ÚåÔÐèÒ¶( disk.nvme0c0n1.out‚84439.80029296875 disk.sda3.out‚ 441.1640625 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0* disk./.usagePercent‚19.387744976681606 disk.rbd1.out‚ 14.3212890625 disk.nvme0c0n1.in‚472.125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 cpu‚25.27457830103759 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda2.in‚0 disk.nvme0n1.in‚472.125 disk.sda3.in‚0.203125 network.sent‚ 3.0169902e+07 disk.drbd2266.in‚0# disk.dm-0.out‚84439.80029296875 disk.drbd1716.in‚0 disk.sda.out‚ 441.1640625 disk.rbd0.in‚0 disk.dm-1.out‚ 441.1640625 proc.cpu.threads‚17& disk.nvme0n1.out‚84439.80029296875( proc.memory.availableMB‚ 312.83203125 disk.drbd1716.out‚0 disk.drbd2265.out‚0$ disk./.usageGB‚288.8914108276367 disk.dm-0.in‚472.125$ memory_percent‚98.09062480926514# proc.memory.rssMB‚ 2892.84765625* proc.memory.percent‚17.656540870666504 network.recv‚ 4.004651e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0ßö;4ä,/ _wandbruntime‚11814  _runtime‚11814åtÉå,à & lr-AdamW/pg2‚4.953394325849984e-07 trainer/global_step‚11749 % _timestamp‚1.7884398359948874e+09 _runtime‚11814.349938628 _step‚963 & lr-AdamW/pg1‚4.953394325849984e-07õHù’Îæ,È _runtime‚11814.349938628 _step‚963 & lr-AdamW/pg1‚4.953394325849984e-07 & lr-AdamW/pg2‚4.953394325849984e-07 trainer/global_step‚11749 % _timestamp‚1.7884398359948874e+09ÃúÍľ4ç,/ _wandbruntime‚11814  _runtime‚11814ºüm$(è,žP \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1398375627471872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252966931 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1593682951624951 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25061263889655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 463546308 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 252990037 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 463569420 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125956212 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1568621687735296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚92265189 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1593686080521843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34807535489536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1620999704726528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1595895991238656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚25106818881536 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1620999704726528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1568621713163264 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚11749 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25106818881536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125956215 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1595896017630208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1363568177228288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1595896017630208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218221404 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210579377 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1595895991238656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218220854 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210579377 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1621002836511744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚92264642 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34807478377984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1621002836511744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456X¥§Ï é,£P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1593682951624951 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25061263889655 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 463546308 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 252990037 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 463569420 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125956212 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1568621687735296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚92265189 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1593686080521843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34807535489536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1620999704726528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1595895991238656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚25106818881536 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1620999704726528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1568621713163264 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚11749 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25106818881536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125956215 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159589601763È¿œ‰0208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1363568177228288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1595896017630208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218221404 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210579377 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1595895991238656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218220854 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210579377 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1621002836511744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚92264642 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34807478377984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1621002836511744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚85245952 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1398375627471872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252966931 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302ÄrÌ|å4ê,/ _wandbruntime‚11815  _runtime‚11815laÐ.'ë,¨N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1398491036358144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 463607820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚965 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1621133311128576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25108931208704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 463607820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596027485326848 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1363680588092416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210596661 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210596667 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1363680559958016 % _timestamp‚1.7884398368896449e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252988021 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25105825801728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚92272135 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1596027485326848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218238600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _runtime‚11815.245216492 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1621136442927104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125966465 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25063372682755 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11749ר€Ð3'ì,­N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125966465 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25063372682755 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1398491036358144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 463607820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 _step‚965 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1621133311128576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25108931208704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 463607820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596027485326848 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1363680588092416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210596661 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210596667 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1363680559958016 % _timestamp‚1.7884398368896449e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252988021 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25105825801728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚92272135 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1596027485326848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218238600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚59195392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _runtime‚11815.245216492 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1621136442927104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 Ebló,‚(:á °ÚåÔÃ…¸ disk.drbd2266.in‚0 disk.nvme0c0n1.in‚472.125% disk./.usageGB‚288.90870666503906 disk.drbd1716.out‚0% disk.nvme0n1.out‚84508.6728515625$ memory_percent‚98.09098243713379 network.recv‚ 4.014953e+06' proc.memory.availableMB‚ 312.7734375 cpu‚25.273982962033458 disk.drbd2265.out‚0 network.sent‚ 3.0284369e+07 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.sda3.in‚0.203125 disk.drbd2080.in‚0' disk.nvme0c0n1.out‚84508.6728515625 disk.dm-0.in‚472.125 disk.sda1.out‚0# proc.memory.rssMB‚ 2892.84765625 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 14.4697265625 disk.sda3.out‚ 442.22265625* proc.memory.percent‚17.656540870666504 proc.cpu.threads‚17) disk./.usagePercent‚19.38890571484185 disk.nvme0n1.in‚472.125 disk.rbd1.in‚0" disk.dm-0.out‚84508.6728515625 disk.drbd2080.out‚0 disk.sda.out‚ 442.22265625 disk.dm-1.out‚ 442.22265625 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.203125Æ*÷Ážô,‚(:“ ¿ÚåÔݾ«µ gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚258.121' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.624199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀTÄÏpõ,‚(:å ¿Úåԙ̽¶ disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 14.5634765625 disk.nvme0n1.in‚472.125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚19.387780561728864 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚472.125 disk.sda3.out‚ 443.33203125 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.273707014296182 network.sent‚ 3.0290379e+07& disk.nvme0n1.out‚84550.20947265625 disk.dm-1.out‚ 443.33203125 disk.sda1.out‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚84550.20947265625 disk.sda.in‚0.203125# proc.memory.rssMB‚ 2892.84765625# disk.dm-0.out‚84550.20947265625 disk.drbd2265.out‚0$ memory_percent‚98.09269905090332 network.recv‚ 4.019345e+06 disk.dm-0.in‚472.125 disk.sda3.in‚0.203125 disk.sda.out‚ 443.33203125 disk.drbd2265.in‚0' proc.memory.availableMB‚ 312.4921875* proc.memory.percent‚17.656540870666504% disk./.usageGB‚288.89194107055664¿ä߶4ö,/ _wandbruntime‚11861  _runtime‚11861”ìYÊ÷,Ä ' lr-AdamW/pg1‚3.6080742862429955e-07 ' lr-AdamW/pg2‚3.6080742862429955e-07 trainer/global_step‚11799 $ _timestamp‚1.788439883053345e+09 _runtime‚11861.408712143 _step‚967Áœ”OÏø,É ' lr-AdamW/pg1‚3.6080742862429955e-07 ' lr-AdamW/pg2‚3.6080742862429955e-07 trainer/global_step‚11799 $ _timestamp‚1.788439883053345e+09 _runtime‚11861.408712143 _step‚967ÇX·¶J4ù,/ _wandbruntime‚11861  _runtime‚11861äY6_ú,žP OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1602470695648256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1627683171027968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1602470695648256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211443577 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219114520 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚92645658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34953410287104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 211443577 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34953465301504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25209343594496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 465491472 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1600261383828851 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1602470722039808 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126468865 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1404142102587392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1627680039242752 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1404142185736192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1369188720434688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211443583 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211443583 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1369188692300288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1575091563224064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1627683171027968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254047889 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 219115093 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 trainer/global_step‚11799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1627680039242752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚25209343594496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚92646228 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25166717135863 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1602470722039808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25212448988160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25166717135863 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1602470722039808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25212448988160 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1602470695648256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1627683171027968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1602470695648256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211443577 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219114520 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚92645658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚34953410287104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 211443577 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚34953465301504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25209343594496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 465491472 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1600261383828851 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1602470722039808 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126468865 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1404142102587392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1627680039242752 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1404142185736192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1369188720434688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211443583 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211443583 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1369188692300288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1575091563224064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1627683171027968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254047889 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 219115093 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 trainer/global_step‚11799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864È\U†ü4ü,/ _wandbruntime‚11862  _runtime‚11862Qêö].'ý,¨N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1627813645644800 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1602602189736448 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1369301131298816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 465529872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚11799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1404257531832320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 465529872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1602602216128000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211460861 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254069005 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34956400533504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1575220934797312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1600389760726275 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25214561315328 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219132215 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1627816777443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1600392889623375 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚92653650 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884398839544322e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 _runtime‚11862.310046534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1627816777443328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254069005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126479115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25211455908352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 465506734 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211460867 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 465529872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 trainer/global_step‚11799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1404257531832320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 465529872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1602602216128000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211460861 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254069005 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚34956400533504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMoï¾n¦nitor.on_train_batch_end/requested_bytes.large_pool.freed‚1575220934797312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1600389760726275 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25214561315328 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219132215 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1627816777443328 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1600392889623375 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚92653650 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 % _timestamp‚1.7884398839544322e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 _runtime‚11862.310046534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1627816777443328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254069005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126479115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25211455908352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 465506734 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211460867 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25318079094784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255105741 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25272170382071 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1634363505544192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212307783 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1634360373758976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1634360373758976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1609045400057856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126981515 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126981512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚93022922 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35099467518464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1374809263641088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35099522532864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1606836687135859 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25314973701120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212307783 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1581561387856896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 255082635 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212307777 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 _runtime‚11908.772289366 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 $ _timestamp‚1.788439930417013e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1374809235506688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1606833558238967 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚25275273850995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 trainer/global_step‚11849—>ØùÌ-¢P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255105741 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25272170382071 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1634363505544192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212307783 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1634360373758976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1634360373758976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1609045400057856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126981515 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126981512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚93022922 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35099467518464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1374809263641088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35099522532864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1606836687135859 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25314973701120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212307783 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1581561387856896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 255082635 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212307777 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 _runtime‚11908.772289366 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 $ _timestamp‚1.788439930417013e+09 G>DeviceStatsMonit«š†“\or.on_train_batch_start/active.large_pool.peak‚3329 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1374809235506688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1606833558238967 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚25275273850995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 trainer/global_step‚11849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220003887 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1609045426449408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 467413524 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1634363505544192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚93022375 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚83148800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1409908786173952 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1409908703025152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1609045426449408 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255082635 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25318079094784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180ÌÓ‡S)4Ž-/ _wandbruntime‚11909  _runtime‚11909y¾;-'-§N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1606965064033283 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25320191421952 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1609176920537600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255103725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126991765 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚35102379628032 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1634493980161024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126991768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255103725 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1609176894146048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚93030256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1634497111959552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220021487 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467428786 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467451924 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1374921674505216 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1374921646370816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚11849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25317086014976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25317086014976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25274279175171 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25320191421952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1581690810286080 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212325061 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212325061 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 467428786 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1410024111231488 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1581690784858112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212325067 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976R9+Ó2'-¬N EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220021487 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467428786 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467451924 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1374921674505216 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1374921646370816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 trainer/global_step‚11849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25317086014976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25317086014976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25274279175171 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25320191421952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1581690810286080 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212325061 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212325061 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 467428786 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1410024111231488 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1581690784858112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212325067 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1606965064033283 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25320191421952 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1609176920537600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255103725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126991765 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚35102379628032 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1634493980161024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126991768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255103725 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1609176894146048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚93030256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1634497111959552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 F¼ŠÇa’-Û 3loss/FBt1/assignment_loss‚0.0011169977519272488 0loss/SRqqt4/detection_loss‚2.461552131634492 /loss/FBt1/detection_loss‚0.6602210741962345 /loss/FRt4/assignment_loss‚4.231394604400352 2loss/SRqqt4/assignment_loss‚1.5782275603940854 1loss/SRqqt3/detection_loss‚1.6816172313015416 _runtime‚11909.69963354 %loss/total_loss‚39.36565204307146 0loss/FRt1/assignment_loss‚1.5403931736946106 1loss/FBt4/assignment_loss‚0.13309673919138731 _step‚974 2loss/SRqqt3/assignment_loss‚1.1121951315622285 1loss/FBt3/assignment_loss‚0.07285351239676986 /loss/FBt4/detection_loss‚1.7548643818923406 0loss/FRt2/assignment_loss‚2.9645260350648748 /loss/FRt1/detection_loss‚1.4770710932138233 .loss/FRt4/detection_loss‚3.108731989507322 $ _timestamp‚1.788439931344671e+09 2loss/SRqqt1/assignment_loss‚1.0219299913362854 2loss/SRqqt2/assignment_loss‚1.4317777083979712 -loss/FBt2/detection_loss‚2.14166616068946 epoch‚9 trainer/global_step‚11849 0loss/FRt3/assignment_loss‚2.3968090373416278 2loss/FBt2/assignment_loss‚0.051594282060654625 /loss/FRt2/detection_loss‚2.2237157890963015 /loss/FRt3/detection_loss‚2.1780562359233233 1loss/SRqqt1/detection_loss‚0.6094604243355041 1loss/SRqqt2/detection_loss‚1.8973433415095011 /loss/FBt3/detection_loss‚2.6354374161788394Ï-Üwf“-à 2loss/SRqqt3/assignment_loss‚1.1121951315622285 1loss/FBt3/assignment_loss‚0.07285351239676986 /loss/FBt4/detection_loss‚1.7548643818923406 0loss/FRt2/assignment_loss‚2.9645260350648748 /loss/FRt1/detection_loss‚1.4770710932138233 .loss/FRt4/detection_loss‚3.108731989507322 $ _timestamp‚1.788439931344671e+09 2loss/SRqqt1/assignment_loss‚1.0219299913362854 2loss/SRqqt2/assignment_loss‚1.4317777083979712 -loss/FBt2/detection_loss‚2.14166616068946 epoch‚9 trainer/global_step‚11849 0loss/FRt3/assignment_loss‚2.3968090373416278 2loss/FBt2/assignment_loss‚0.051594282060654625 /loss/FRt2/detection_loss‚2.2237157890963015 /loss/FRt3/detection_loss‚2.1780562359233233 1loss/SRqqt1/detection_loss‚0.6094604243355041 1loss/SRqqt2/detection_loss‚1.8973433415095011 /loss/FBt3/detection_loss‚2.6354374161788394 3loss/FBt1/assignment_loss‚0.0011169977519272488 0loss/SRqqt4/detection_loss‚2.461552131634492 /loss/FBt1/detection_loss‚0.6602210741962345 /loss/FRt4/assignment_loss‚4.231394604400352 2loss/SRqqt4/assignment_loss‚1.5782275603940854 1loss/SRqqt3/detection_loss‚1.6816172313015416 _runtime‚11909.69963354 %loss/total_loss‚39.36565204307146 0loss/FRt1/assignment_loss‚1.5403931736946106 1loss/FBt4/assignment_loss‚0.13309673919138731 _step‚974Îù€°§“”-‚(:ˆ ûÚåÔáõ¢µ gpu.0.gpu‚70.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚287.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F˜ý©b•-‚(:× ûÚåÔÇû· disk.drbd1716.out‚0 disk.drbd2080.in‚0 network.sent‚ 3.0416957e+07 disk.sda2.out‚0 disk.dm-1.out‚ 445.421875# proc.memory.availableMB‚686.125 cpu‚25.272558654944678$ disk./.usageGB‚288.9021530151367 disk.sda2.in‚0 disk.rbd1.out‚ 14.751953125 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.65751838684082* disk./.usagePercent‚19.388465893898026 disk.dm-1.in‚0.203125# disk.dm-0.out‚84731.16748046875" proc.memory.rssMB‚ 2893.0078125 disk.rbd1.in‚0 disk.dm-0.in‚472.125 disk.sda3.out‚ 445.421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚472.125$ memory_percent‚95.81222534179688 network.recv‚ 4.038082e+06 disk.nvme0n1.in‚472.125& disk.nvme0n1.out‚84731.16748046875( disk.nvme0c0n1.out‚84731.16748046875 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.sda.out‚ 445.421875 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0.203125W“–-‚(:ˆ ŠÛåÔ®¨ï´ gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚292.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dw^Ñg—-‚(:Ü ŠÛåÔÁ·É· disk.nvme0c0n1.in‚472.125 disk.dm-0.in‚472.125 disk.drbd2080.in‚0 disk.sda.in‚0.203125 network.sent‚ 3.0525642e+07 disk.drbd1716.in‚0 disk.sda.out‚ 446.00390625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.814453125! disk.dm-0.out‚84775.431640625 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚472.125$ disk.nvme0n1.out‚84775.431640625' proc.memory.availableMB‚ 686.8671875 network.recv‚ 4.043998e+06 disk.sda1.in‚0 disk.drbd2080.out‚0$ memory_percent‚95.80769538879395 disk.dm-1.in‚0.203125 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.272277393296594* disk./.usagePercent‚19.388855793372674 disk.dm-1.out‚ 446.00390625 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚84775.431640625 proc.cpu.threads‚17 disk.sda3.in‚0.203125 disk.sda3.out‚ 446.00390625" proc.memory.rssMB‚ 2893.0078125) proc.memory.percent‚17.65751838684082% disk./.usageGB‚288.90796279907227¡ W|“˜-‚(:ˆ ™ÛåÔâÖñ´ gpu.0.gpu‚91.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚47.0 gpu.0.powerWatts‚244.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¤¦j™-‚(:ß ™ÛåÔñ®â·% disk./.usageGB‚288.91726303100586( disk.nvme0c0n1.out‚84826.71630859375( proc.memory.availableMB‚ 686.62109375 disk.dm-0.in‚472.125 disk.sda3.in‚0.203125 network.sent‚ 3.0531922e+07 disk.rbd0.in‚0 disk.nvme0n1.in‚472.125 disk.nvme0c0n1.in‚472.125 disk.sda.in‚0.203125$ memory_percent‚95.80919742584229* disk./.usagePercent‚19.389479939741154 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 446.5859375 disk.rbd1.out‚ 14.814453125 disk.sda1.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚84826.71630859375 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2893.0078125) proc.memory.percent‚17.65751838684082 cpu‚25.27201878700218 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.dm-1.out‚ 446.5859375 disk.drbd2265.out‚0# disk.dm-0.out‚84826.71630859375 disk.drbd2265.in‚0 disk.sda3.out‚ 446.5859375 disk.drbd2080.in‚0 network.recv‚ 4.048678e+06„­ö “š-‚(:ˆ ¨ÛåÔ¬½‘µ gpu.0.gpu‚50.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚279.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ƒÉÙ.›-‚(:á ¨ÛåÔôÏì· disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 14.8466796875 disk.sda3.in‚0.203125 network.recv‚ 4.052142e+06) disk./.usagePercent‚19.39021519337949 disk.sda.out‚ 447.03515625 disk.nvme0n1.in‚472.125( disk.nvme0c0n1.out‚84865.14599609375% disk./.usageGB‚288.92821884155273 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 686.84765625 disk.nvme0c0n1.in‚472.125 disk.dm-0.in‚472.125# disk.dm-0.out‚84865.14599609375 disk.sda3.out‚ 447.03515625 disk.drbd1716.out‚0 disk.drbd2080.out‚0) proc.memory.percent‚17.65751838684082 disk.sda2.out‚0 disk.dm-1.in‚0.203125 disk.sda1.in‚0# memory_percent‚95.8078145980835& disk.nvme0n1.out‚84865.14599609375 disk.dm-1.out‚ 447.03515625 disk.drbd2266.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2893.0078125 proc.cpu.threads‚17 network.sent‚ 3.0537233e+07 disk.sda.inЃNY>‚0.203125 disk.drbd1716.in‚0 cpu‚25.27171959158767,%è{4œ-/ _wandbruntime‚11955  _runtime‚11955B¤|ÜÈ- trainer/global_step‚11899 $ _timestamp‚1.788439977449483e+09 _runtime‚11955.804531377 _step‚975 & lr-AdamW/pg1‚1.554897106483743e-07 & lr-AdamW/pg2‚1.554897106483743e-07•-½±Íž-Ç _runtime‚11955.804531377 _step‚975 & lr-AdamW/pg1‚1.554897106483743e-07 & lr-AdamW/pg2‚1.554897106483743e-07 trainer/global_step‚11899 $ _timestamp‚1.788439977449483e+09ÏØ7Úþ4Ÿ-/ _wandbruntime‚11955  _runtime‚11955ªõuã#( -P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3878632448 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1615620130859008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25423709201408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1641040708275200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚25420603807744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1641043840060416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1641040708275200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 $ _timestamp‚1.788439977450225e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127494165 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35245458195968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256163593 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220895096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚93400934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1588031263345664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25420603807744 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1615620104467456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220895648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1615620104467456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256140487 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213171977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚25423709201408 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1615620130859008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1641040708275200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 $ _timestamp‚1.788439977450225e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127494165 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35245458195968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256163593 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220895096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚93400934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1588031263345664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25420603807744 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1615620104467456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220895648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1615620104467456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256140487 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213171977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚25423709201408 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1615620130859008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1641040708275200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚25420603807744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1641043840060416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220912859 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1415790560275456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469373976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1613540367340291 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚93408949 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11956.702156781 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25382835890511 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220912859 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1415790560275456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469373976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1613540367340291 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚93408949 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11956.702156781 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25382835890511 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚25486180343411 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1622194808877056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚25526233914368 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 257221445 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471234516 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1622194835268608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35391317391360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚93777851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1619987293749875 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1386050321919488 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1647721042791424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1622194808877056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1421441580102144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1619984164852983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257221445 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35391258182656 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214036183 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 257198339 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214036183 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214036177 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664v×>%³-¢P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚25486180343411 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1622194808877056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚25526233914368 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 257221445 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3938451456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471234516 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1622194835268608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35391317391360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚93777851 \ODeviceStatsMonitor.on_train_ba=r×ßtch_start/inactive_split_bytes.small_pool.current‚59208704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1619987293749875 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1386050321919488 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1647721042791424 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1622194808877056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1421441580102144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27964 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1619984164852983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257221445 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35391258182656 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 21432077312 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214036183 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 257198339 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214036183 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214036177 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128006812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257198339 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚25483076874487 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878ÔÊ’R@4´-/ _wandbruntime‚12003  _runtime‚12003„Z?-'µ-§N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25531451635200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1620118799544399 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471296028 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257219429 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚93785996 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214053461 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128017068 % _timestamp‚1.7884400253792284e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1386162732783616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚35394233741312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1647857780992000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1622326302965248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25528346228224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471272890 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1622326329356800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1622326302965248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1647854649193472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25488289136719 trainer/global_step‚11949 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25528346228224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128017065 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471272890 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1620115670647299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1594630484979712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25485185667587 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _runtime‚12003.73477473 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214053467 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1647857780992000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221803064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221802530 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257219429 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 257242561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1594630510407680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1622326329356800 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚35394176643072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1386162760918016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221802530 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257219429 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 257242561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1594630510407680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1622326329356800 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚35394176643072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1386162760918016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471296028 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257219429 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚93785996 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214053461 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128017068 % _timestamp‚1.7884400253792284e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1386162732783616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚35394233741312 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1647857780992000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1622326302965248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25528346228224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471272890 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1622326329356800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1622326302965248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1647854649193472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25488289136719 trainer/global_step‚11949 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚25528346228224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128017065 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471272890 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1620115670647299 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1594630484979712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25485185667587 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 _runtime‚12003.73477473 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214053467 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1647857780992000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221803064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q€`Å4·-/ _wandbruntime‚12003  _runtime‚12003XXw•]¸-× 2loss/SRqqt1/assignment_loss‚0.8660688684015584 1loss/FBt3/assignment_loss‚0.07673098880981012 %loss/total_loss‚38.56621026230833 trainer/global_step‚11949 _runtime‚12003.735952132 1loss/FBt4/assignment_loss‚0.06613747645381161 $ _timestamp‚1.788440025381014e+09 0loss/FRt1/assignment_loss‚1.1875351776679357 2loss/FBt1/assignment_loss‚6.32562477353555e-05 0loss/FRt3/assignment_loss‚3.1504081073734493 2loss/SRqqt2/assignment_loss‚1.2686708089378145 2loss/SRqqt4/assignment_loss‚1.3288101801234815 /loss/FRt1/detection_loss‚1.4369345407932996 0loss/SRqqt1/detection_loss‚1.041881670489123 _step‚982 2loss/SRqqt3/assignment_loss‚1.1632324101713796 1loss/FBt2/assignment_loss‚0.07461494502786081 .loss/FRt4/detection_loss‚2.810362632274628 /loss/FBt3/detection_loss‚2.2075044187632473 /loss/FRt2/assignment_loss‚2.981984430551529 /loss/FRt4/assignment_loss‚2.597353401184082 /loss/FRt2/detection_loss‚2.4982261538505552 0loss/SRqqt2/detection_loss‚2.515953578330852 1loss/SRqqt3/detection_loss‚1.5117036135246356 /loss/FBt2/detection_loss‚2.0107626020908356 epoch‚9 /loss/FBt1/detection_loss‚0.6258315402542202 /loss/FBt4/detection_loss‚2.4458838267759844 -loss/FRt3/detection_loss‚2.43552809746729 0loss/SRqqt4/detection_loss‚2.264027536743217ëEb¹-Ü 2loss/SRqqt1/assignment_loss‚0.8660688684015584 1loss/FBt3/assignment_loss‚0.07673098880981012 %loss/total_loss‚38.56621026230833 trainer/global_step‚11949 _runtime‚12003.735952132 1loss/FBt4/assignment_loss‚0.06613747645381161 $ _timestamp‚1.788440025381014e+09 0loss/FRt1/assignment_loss‚1.1875351776679357 2loss/FBt1/assignment_loss‚6.32562477353555e-05 0loss/FRt3/assignment_loss‚3.1504081073734493 2loss/SRqqt2/assignment_loss‚1.2686708089378145 2loss/SRqqt4/assignment_loss‚1.3288101801234815 /loss/FRt1/detection_loss‚1.4369345407932996 0loss/SRqqt1/detection_loss‚1.041881670489123 _step‚982 2loss/SRqqt3/assignment_loss‚1.1632324101713796 1loss/FBt2/assignment_loss‚0.07461494502786081 .loss/FRt4/detection_loss‚2.810362632274628 /loss/FBt3/detection_loss‚2.2075044187632473 /loss/FRt2/assignment_loss‚2.981984430551529 /loss/FRt4/assignment_loss‚2.597353401184082 /loss/FRt2/detection_loss‚2.4982261538505552 0loss/SRqqt2/detection_loss‚2.515953578330852 1loss/SRqqt3/detection_loss‚1.5117036135246356 /loss/FBt2/detection_loss‚2.0107626020908356 epoch‚9 /loss/FBt1/detection_loss‚0.6258315402542202 /loss/FBt4/detection_loss‚2.4458838267759844 -loss/FRt3/detection_loss‚2.43552809746729 0loss/SRqqt4/detection_loss‚2.264027536743217ÖcäóÓ[º-‚(:Ð äÛåÔ™ˆÒ¶ network.sent‚ 3.0663712e+07 network.recv‚ 4.07087e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚85068.86376953125 disk.sda2.in‚0) disk./.usagePercent‚19.39184416931986 disk.sda3.in‚0.203125 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 15.21484375$ disk./.usageGB‚288.9524917602539 disk.nvme0c0n1.in‚472.125# disk.dm-0.out‚85068.86376953125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚472.125 disk.dm-1.in‚0.203125 disk.sda3.out‚449.375 disk.drbd2266.in‚0$ memory_percent‚95.81029415130615" proc.memory.rssMB‚ 2893.0078125 disk.sda1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 686.44140625 disk.dm-1.out‚449.375 disk.drbd2266.out‚0) proc.memory.percent‚17.65751838684082 cpu‚25.270619996958388 disk.sda.out‚449.375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚85068.86376953125 disk.dm-0.in‚472.125 disk.drbd2265.out‚0oÙ“»-‚(:ˆ äÛåÔ¨Ëô¶ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚299.624' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú¤Uïž¼-‚(:“ óÛåÔŽ‡å´ gpu.0.gpu‚76.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚273.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.635400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÆÍh½-‚(:Ý óÛåÔí±·) proc.memory.percent‚17.65751838684082 proc.cpu.threads‚17 network.sent‚ 3.0772518e+07* disk./.usagePercent‚19.393193073017773% disk.nvme0n1.out‚85099.6943359375 disk.dm-1.out‚ 450.18359375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚472.125 disk.sda3.out‚ 450.18359375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0$ memory_percent‚95.81177234649658 disk.nvme0n1.in‚472.125 disk.dm-0.in‚472.125 disk.rbd0.out‚0$ disk./.usageGB‚288.9725914001465 disk.sda2.out‚0 disk.dm-1.in‚0.203125" disk.dm-0.out‚85099.6943359375 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2893.0078125' disk.nvme0c0n1.out‚85099.6943359375( proc.memory.availableMB‚ 686.19921875 cpu‚25.270321552655194 network.recv‚ 4.07711e+06 disk.sda3.in‚0.203125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 15.30859375 disk.sda.out‚ 450.18359375 disk.drbd2265.in‚0Îð社-‚(:’ ‚ÜåÔÜç’µ gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚279.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bh«c¿-‚(:Ø ‚ÜåÔ¡Òì¶ disk.nvme0c0n1.in‚472.125 disk.dm-0.in‚472.125 disk.sda.in‚0.203125% disk.nvme0n1.out‚85144.0341796875 disk.drbd2265.in‚0 disk.drbd2266.in‚0" disk.dm-0.out‚85144.0341796875 disk.sda1.out‚0$ proc.memory.availableMB‚686.4375 network.recv‚ 4.080786e+06) disk./.usagePercent‚19.39359142074105$ disk./.usageGB‚288.9785270690918 disk.sda.out‚ 451.01171875 disk.nvme0n1.in‚472.125 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd1.out‚ 15.55078125 disk.sda3.out‚ 451.01171875 cpu‚25.270050032253057 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.203125) proc.memory.percent‚17.65751838684082 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚85144.0341796875 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0$ memory_percent‚95.81031799316406" proc.memory.rssMB‚ 2893.0078125 network.sent‚ 3.077843e+07 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 451.011718759…nc4À-/ _wandbruntime‚12050  _runtime‚12050å­1ÝÉÁ-à trainer/global_step‚11999 % _timestamp‚1.7884400717198813e+09 _runtime‚12050.074919806 _step‚983 & lr-AdamW/pg1‚3.530228246773177e-08 & lr-AdamW/pg2‚3.530228246773177e-08hFœÎÂ-È & lr-AdamW/pg1‚3.530228246773177e-08 & lr-AdamW/pg2‚3.530228246773177e-08 trainer/global_step‚11999 % _timestamp‚1.7884400717198813e+09 _runtime‚12050.074919806 _step‚983×Í—\æ4Ã-/ _wandbruntime‚12050  _runtime‚12050éw8'Ä-P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.onË™'_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚94157528 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1391670865125888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 222677561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1626562597056883 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12050.075868176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25631864020992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚25591633589619 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1654401377307648 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1427207746445312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1628769513286656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258279297 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214900383 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35536853185024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1427207659102208 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1391670893260288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258279297 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214900377 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚94158096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 trainer/global_step‚11999 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222676990 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1600970963467264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473179680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 $ _timestamp‚1.788440071720611e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1654404509092864 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1626559468159991 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1654401377307648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35536793976320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214900383 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1628769539678208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25634969414656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984q5ü‹((Å-¢P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1878 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 21880356864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚971 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚94157528 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1391670865125888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 222677561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1626562597056883 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12050.075868176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25631864020992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2935639552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚25591633589619 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1654401377307648 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1427207746445312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1628769513286656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258279297 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23106 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1878 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214900383 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35536853185024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1427207659102208 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1391670893260288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258279297 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214900377 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚94158096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚59208704 trainer/global_step‚11999 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222676990 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1600970963467264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23106 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473179680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 $ _timestamp‚1.788440071720611e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1654404509092864 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1626559468159991 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1654401377307648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚35536793976320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214900383 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1628769539678208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚25634969414656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 24572329984ئ{²4Æ-/ _wandbruntime‚12050  _runtime‚12050­u .'Ç-¨N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473194942 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25593742382927 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1654538115508224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25637081741824 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25590638913795 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1628901033766400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25637081741824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214917661 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222694906 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1427323058430976 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1427322971101184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 222695476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚94165191 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1391783304124416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1628901007374848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214917661 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258277281 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1391783275990016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1628901033766400 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128529718 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚94165758 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1628901007374848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473218080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚35539695111168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1601100360468480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚12050.963426777 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214917667 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1601100335040512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473218080 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214917661 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258277281 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1391783275990016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1628901033766400 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128529718 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚94165758 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚87329792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 28510781440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1628901007374848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473218080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚35539695111168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1601100360468480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚12050.963426777 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 21491766CpQý7 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1601100335040512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473218080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214917661 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222694906 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1427323058430976 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1427322971101184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 222695476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚94165191 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1391783304124416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249ÙwõN4É-/ _wandbruntime‚12050  _runtime‚12050­&fYÊ-Ó 1loss/FBt2/assignment_loss‚0.27687913766130234 1loss/FBt4/assignment_loss‚0.04835392613174571 2loss/SRqqt4/assignment_loss‚1.4504220120223432 /loss/FBt1/detection_loss‚0.6504676695771967 trainer/global_step‚11999 .loss/FRt2/assignment_loss‚3.24304899904463 .loss/FRt2/detection_loss‚2.479204034805298 /loss/FBt4/detection_loss‚2.0371213455994925 $ _timestamp‚1.788440072609614e+09 1loss/SRqqt3/assignment_loss‚0.924984495567738 1loss/SRqqt2/detection_loss‚2.5194317813103018 &loss/total_loss‚40.171937791385496 epoch‚9 0loss/FBt1/assignment_loss‚0.0408776290898964 .loss/FRt3/detection_loss‚2.202582287075727 .loss/FBt2/detection_loss‚2.069900614023209 0loss/FRt4/assignment_loss‚3.6868144683539867 0loss/FBt3/assignment_loss‚0.1513923896709457 .loss/FRt1/detection_loss‚1.475326097512152 0loss/SRqqt1/detection_loss‚1.071544800169658 0loss/FRt3/assignment_loss‚1.9157991124236065 1loss/SRqqt1/assignment_loss‚1.081487136380072 2loss/SRqqt2/assignment_loss‚1.5547206631073585 1loss/SRqqt3/detection_loss‚1.4643320963397377 .loss/FBt3/detection_loss‚2.718160092830658 0loss/FRt1/assignment_loss‚1.5897475555539131 /loss/FRt4/detection_loss‚2.9720904752612114 1loss/SRqqt4/detection_loss‚2.5472489718733162 _runtime‚12050.964572419 _step‚986“¦^Ë-Ø 0loss/FRt4/assignment_loss‚3.6868144683539867 0loss/FBt3/assignment_loss‚0.1513923896709457 .loss/FRt1/detection_loss‚1.475326097512152 0loss/SRqqt1/detection_loss‚1.071544800169658 0loss/FRt3/assignment_loss‚1.9157991124236065 1loss/SRqqt1/assignment_loss‚1.081487136380072 2loss/SRqqt2/assignment_loss‚1.5547206631073585 1loss/SRqqt3/detection_loss‚1.4643320963397377 .loss/FBt3/detection_loss‚2.718160092830658 0loss/FRt1/assignment_loss‚1.5897475555539131 /loss/FRt4/detection_loss‚2.9720904752612114 1loss/SRqqt4/detection_loss‚2.5472489718733162 _runtime‚12050.964572419 _step‚986 1loss/FBt2/assignment_loss‚0.27687913766130234 1loss/FBt4/assignment_loss‚0.04835392613174571 2loss/SRqqt4/assignment_loss‚1.4504220120223432 /loss/FBt1/detection_loss‚0.6504676695771967 trainer/global_step‚11999 .loss/FRt2/assignment_loss‚3.24304899904463 .loss/FRt2/detection_loss‚2.479204034805298 /loss/FBt4/detection_loss‚2.0371213455994925 $ _timestamp‚1.788440072609614e+09 1loss/SRqqt3/assignment_loss‚0.924984495567738 1loss/SRqqt2/detection_loss‚2.5194317813103018 &loss/total_loss‚40.171937791385496 epoch‚9 0loss/FBt1/assignment_loss‚0.0408776290898964 .loss/FRt3/detection_loss‚2.202582287075727 .loss/FBt2/detection_loss‚2.069900614023209ڸ톮“Ì-‚(:ˆ ‘ÜåÔÿôíµ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚296.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.2754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¨:gÍ-‚(:Ü ‘ÜåÔ¼”ì¶ disk.dm-1.out‚ 451.7421875 disk.sda3.out‚ 451.7421875 disk.drbd2080.out‚0 disk.sda.in‚0.203125 disk.rbd0.out‚0 disk.nvme0n1.in‚472.125 disk.sda2.in‚0 disk.dm-0.in‚472.125) proc.memory.percent‚17.65751838684082 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0" disk.dm-0.out‚85186.7138671875 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚95.80914974212646 proc.cpu.threads‚17 network.sent‚ 3.0784727e+07% disk./.usageGB‚288.97936630249023 disk.dm-1.in‚0.203125 disk.sda1.in‚0 disk.drbd1716.in‚0 network.recv‚ 4.085344e+06* disk./.usagePercent‚19.393647742398585 disk.nvme0c0n1.in‚472.125 disk.sda.out‚ 451.7421875 disk.drbd2265.out‚0( proc.memory.availableMB‚ 686.62890625 disk.rbd1.in‚0 disk.rbd1.out‚ 15.55078125 disk.sda3.in‚0.203125 disk.drbd1716.out‚0% disk.nvme0n1.out‚85186.7138671875 disk.sda2.out‚0" proc.memory.rssMB‚ 2893.0078125 cpu‚25.269797629004348' disk.nvme0c0n1.out‚85186.71386718753Ù*ÏÎ-‚(:’  ÜåÔåÛ†µ gpu.0.gpu‚68.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.135' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.626999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~`‘iÏ-‚(:Þ  ÜåÔ²¦Õ· disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 452.09765625 disk.sda2.out‚0 disk.drbd2266.in‚0$ memory_percent‚95.81067562103271" proc.memory.rssMB‚ 2893.0078125 disk.drbd2080.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚17.65751838684082 disk.sda1.out‚0 disk.dm-1.in‚0.203125 disk.nvme0c0n1.in‚472.125 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚472.125 disk.sda2.in‚0 network.sent‚ 3.089295e+07 disk.sda3.in‚0.203125 disk.sda1.in‚0' disk.nvme0c0n1.out‚85226.4716796875 network.recv‚ 4.09076e+06% disk./.usageGB‚288.99474334716797 disk.sda3.out‚ 452.09765625 disk.drbd1716.out‚0 disk.sda.in‚0.203125 disk.drbd2265.out‚0 disk.dm-1.out‚ 452.09765625 disk.rbd1.out‚ 15.5830078125* disk./.usagePercent‚19.394679708769104 disk.rbd0.out‚0" disk.dm-0.out‚85226.4716796875 cpu‚25.26950097820308% disk.nvme0n1.out‚85226.4716796875 disk.dm-0.in‚472.125( proc.memory.availableMB‚ 686.37890625)›žÐ-‚(:“ ¯ÜåÔ඲µ gpu.0.gpu‚86.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚48.0 gpu.0.powerWatts‚305.179' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.035799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢|ÂÁcÑ-‚(:Ø ¯ÜåÔœ‰›· disk.nvme0n1.in‚472.125 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚472.125$ memory_percent‚95.81222534179688) proc.memory.percent‚17.65751838684082 cpu‚25.26923115264136 disk.rbd1.in‚0 disk.rbd1.out‚ 15.7626953125# disk./.usageGB‚289.000919342041 disk.dm-1.in‚0.203125( disk.nvme0c0n1.out‚85276.07080078125 disk.drbd1716.in‚0 proc.cpu.threads‚17 network.recv‚ 4.094892e+06 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0# proc.memory.availableMB‚686.125 disk.sda.in‚0.203125 disk.drbd2265.out‚0& disk.nvme0n1.out‚85276.07080078125 disk.nvme0c0n1.in‚472.125 disk.sda3.in‚0.203125 disk.sda3.out‚ 453.0390625" proc.memory.rssMB‚ 2893.0078125 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 453.0390625# disk.dm-0.out‚85276.07080078125 network.sent‚ 3.0898936e+07 disk.sda.out‚ 453.0390625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚19.39509418496704 disk.sda1.out‚0RŒB‡4Ò-/ _wandbruntime‚12097  _runtime‚12097eç(ÉÓ-à _step‚987 & lr-AdamW/pg1‚7.166508598146756e-09 & lr-AdamW/pg2‚7.166508598146756e-09 trainer/global_step‚12049 % _timestamp‚1.7884401188541863e+09 _runtime‚12097.209208521°¥ÇÎÔ-È & lr-AdamW/pg1‚7.166508598146756e-09 & lr-AdamW/pg2‚7.166508598146756e-09 trainer/global_step‚12049 % _timestamp‚1.7884401188541863e+09 _runtime‚12097.209208521 _step‚987Ûh\iœ4Õ-/ _wandbruntime‚12097  _runtime‚12097³r\&$(Ö-žP _runtime‚12097.210150027 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚25737494127616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚57111552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3103468924 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3131785216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 475101732 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1397291408332288 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129032112 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661081711823872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1635344244087808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25737494127616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35682041101312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚94533169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215764577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661081711823872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1633134771466999 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚94533720 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215764583 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1432973392322048 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1607440813528064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 % _timestamp‚1.7884401188549488e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259314043 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1635344217696256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1661084843609088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223565835 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1432973477568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223565281 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27964 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215764577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3296843776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3938451456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 25086444032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661081711823872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1633134771466999 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚94533720 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215764583 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3105393664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 28510781440 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1432973392322048 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3128896892 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1607440813528064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 24572329984 % _timestamp‚1.7884401188549488e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259314043 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1635344217696256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1661084843609088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223565835 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 28510781440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3131785216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 28510781440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚738 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1432973477568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3329 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 24645479249 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 25086444032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223565281 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3876693096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661081711823872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1635344244087808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚25737494127616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚35682041101312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3105393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚94533169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3938451456ÜW|¦ó4Ø-/ _wandbruntime‚12098  _runtime‚12098ÇaUK.'Ù-¨N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1635475738176000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259335133 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 215781861 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1635475711784448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215781867 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚989 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1633269406158415 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1635475711784448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1635475738176000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚35684970437632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475140132 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1397403819196416 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1661218450024448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25742711848448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215781867 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223583119 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1397403847330816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475116994 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1607570185101312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25742711848448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 trainer/global_step‚12049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259358265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259335133 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚94540754 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 % _timestamp‚1.7884401197415304e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475140132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 475116994 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25696092160003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚94541296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1607570210529280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1633266277261315 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25699195629135 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1661218450024448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1433088817768448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25739606441472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129042365 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1661215318225920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215781861 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259358265 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129042368 ¯+3'Ú-­N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3105406976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚85232640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3128897100 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 25086444032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚25742711848448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23132 trainer/global_step‚12049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 28510781440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1878 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1878 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259358265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 425119232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259335133 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 28510781440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3105406976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚94540754 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3876693096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1302 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1878 % _timestamp‚1.7884401197415304e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475140132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 475116994 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 25086444032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚25696092160003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3131798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚94541296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1607570210529280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3103469132 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1633266277261315 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚25699195629135 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 24572329984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚738 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1661218450024448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 21880356864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚57098240 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3131798528 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23132 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1433088817768448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3296843776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚25739606441472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1302 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129042365 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1661215318225920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215781861 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259358265 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129042368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 24572329984 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1635475738176000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259335133 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3938451456 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 24645479249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 215781861 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3938451456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3878632448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1635475711784448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215781867 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2935639552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚989 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1633269406158415 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 21432077312 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1635475711784448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1635475738176000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚35684970437632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3938451456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475140132 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1397403819196416 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1661218450024448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚25742711848448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215781867 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223583119 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1397403847330816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475116994 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1700 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1607570185101312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 24572329984Ý£n”v4Û-/ _wandbruntime‚12098  _runtime‚12098¯Í)x_Ü-Ù 0loss/FRt3/assignment_loss‚2.3119197953492403 /loss/FRt3/detection_loss‚2.6583321169018745 /loss/FBt2/detection_loss‚2.2711650133132935 .loss/FBt3/detection_loss‚2.891399417604719 _runtime‚12098.098415712 0loss/FRt2/assignment_loss‚2.9274669391352957 /loss/FRt1/detection_loss‚1.4582285326971134 epoch‚9 2loss/SRqqt4/assignment_loss‚1.5039927577599883 1loss/FBt3/assignment_loss‚0.10818820926942863 0loss/FBt4/assignment_loss‚0.0541194791757417 /loss/FRt4/detection_loss‚3.1913880006126734 % _timestamp‚1.7884401197434576e+09 2loss/SRqqt2/assignment_loss‚1.3379019662541778 2loss/FBt1/assignment_loss‚0.026009357208636525 1loss/SRqqt1/detection_loss‚1.0602992128494173 -loss/FBt4/detection_loss‚2.22647733038122 trainer/global_step‚12049 _step‚990 0loss/FRt4/assignment_loss‚3.4683521882347437 /loss/FRt2/detection_loss‚2.4103564874428076 /loss/FBt1/detection_loss‚0.6992664186619832 %loss/total_loss‚40.09524761192586 0loss/FBt2/assignment_loss‚0.1598102212854542 0loss/FRt1/assignment_loss‚1.1038033718794165 1loss/SRqqt3/assignment_loss‚1.235685984893092 1loss/SRqqt2/detection_loss‚2.1504282088656175 0loss/SRqqt4/detection_loss‚2.438686966896057 2loss/SRqqt1/assignment_loss‚0.9115081598774427 1loss/SRqqt3/detection_loss‚1.4904614753764251ÄÍêdÝ-Þ 0loss/FRt1/assignment_loss‚1.1038033718794165 1loss/SRqqt3/assignment_loss‚1.235685984893092 1loss/SRqqt2/detection_loss‚2.1504282088656175 0loss/SRqqt4/detection_loss‚2.438686966896057 2loss/SRqqt1/assignment_loss‚0.9115081598774427 1loss/SRqqt3/detection_loss‚1.4904614753764251 0loss/FRt3/assignment_loss‚2.3119197953492403 /loss/FRt3/detection_loss‚2.6583321169018745 /loss/FBt2/detection_loss‚2.2711650133132935 .loss/FBt3/detection_loss‚2.891399417604719 _runtime‚12098.098415712 0loss/FRt2/assignment_loss‚2.9274669391352957 /loss/FRt1/detection_loss‚1.4582285326971134 epoch‚9 2loss/SRqqt4/assignment_loss‚1.5039927577599883 1loss/FBt3/assignment_loss‚0.10818820926942863 0loss/FBt4/assignment_loss‚0.0541194791757417 /loss/FRt4/detection_loss‚3.1913880006126734 % _timestamp‚1.7884401197434576e+09 2loss/SRqqt2/assignment_loss‚1.3379019662541778 2loss/FBt1/assignment_loss‚0.026009357208636525 1loss/SRqqt1/detection_loss‚1.0602992128494173 -loss/FBt4/detection_loss‚2.22647733038122 trainer/global_step‚12049 _step‚990 0loss/FRt4/assignment_loss‚3.4683521882347437 /loss/FRt2/detection_loss‚2.4103564874428076 /loss/FBt1/detection_loss‚0.6992664186619832 %loss/total_loss‚40.09524761192586 0loss/FBt2/assignment_loss‚0.1598102212854542Þ8gÜ žÞ-‚(:“ ¾ÜåÔ¢€³µ gpu.0.gpu‚63.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.129' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.025800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ær¤Åmß-‚(:â ¾ÜåÔŽÅ‹·) proc.memory.percent‚17.65751838684082 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0.203125 disk.drbd2266.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2893.0078125 cpu‚25.268960650438903& disk.nvme0n1.out‚85312.28955078125 disk.dm-1.out‚ 453.69140625 disk.drbd2266.in‚0 network.recv‚ 4.098823e+06* disk./.usagePercent‚19.395961282485512( disk.nvme0c0n1.out‚85312.28955078125 disk.drbd2265.out‚0 disk.rbd0.out‚0# memory_percent‚95.8139181137085 disk.drbd1716.out‚0( proc.memory.availableMB‚ 685.84765625 network.sent‚ 3.0904823e+07 disk.nvme0c0n1.in‚472.125 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚472.125 disk.sda2.in‚0 disk.rbd1.out‚ 15.8251953125 disk.dm-0.in‚472.125# disk.dm-0.out‚85312.28955078125 disk.sda3.in‚0.203125 disk.sda3.out‚ 453.69140625$ disk./.usageGB‚289.0138397216797 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.out‚ 453.69140625 disk.rbd0.in‚0X¿+“à-‚(:ˆ ÍÜåÔÛÒ•µ gpu.0.gpu‚48.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚301.159' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.2318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09õ¸;ká-‚(:à ÍÜåÔ‘£í· network.recv‚ 4.105811e+06 disk.drbd2080.out‚0 cpu‚25.268686920768406 disk.sda1.out‚0 disk.sda.out‚ 454.3828125 disk.sda2.out‚0 disk.dm-1.out‚ 454.3828125( disk.nvme0c0n1.out‚85360.06787109375) disk./.usagePercent‚19.39635169397523 disk.drbd1716.in‚0 disk.rbd1.out‚ 15.8251953125( proc.memory.availableMB‚ 686.33984375 disk.dm-0.in‚472.125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0.203125& disk.nvme0n1.out‚85360.06787109375" proc.memory.rssMB‚ 2893.0078125# disk.dm-0.out‚85360.06787109375 disk.drbd2080.in‚0 disk.nvme0n1.in‚472.125 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ memory_percent‚95.81091403961182 proc.cpu.threads‚17% disk./.usageGB‚289.01965713500977 disk.sda3.out‚ 454.3828125 disk.sda.in‚0.203125 disk.drbd2265.out‚0 network.sent‚ 3.1014179e+07 disk.sda1.in‚0 disk.nvme0c0n1.in‚472.125) proc.memory.percent‚17.65751838684082l¬Wg“â-‚(:ˆ ÜÜåÔ׳µ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚49.0 gpu.0.powerWatts‚298.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.7596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èõËùbã-‚(:× ÜÜåÔúÒÓ¶" proc.memory.rssMB‚ 2893.0078125 network.sent‚ 3.1018586e+07% disk./.usageGB‚289.04426193237305 disk.sda3.in‚0.203125 disk.rbd0.out‚0 disk.drbd2266.in‚0 cpu‚25.268401806083368 disk.nvme0c0n1.in‚472.125( disk.nvme0c0n1.out‚85399.59521484375 disk.dm-0.in‚472.125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0.203125 disk.rbd1.out‚ 15.9736328125 disk.drbd2265.out‚0& disk.nvme0n1.out‚85399.59521484375) proc.memory.percent‚17.65751838684082# disk.dm-0.out‚85399.59521484375 disk.drbd1716.in‚0 disk.sda.out‚454.8125 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.dm-1.out‚454.8125$ memory_percent‚95.81091403961182 disk.sda2.out‚0 network.recv‚ 4.108463e+06 disk.drbd1716.out‚0 disk.nvme0n1.in‚472.125) disk./.usagePercent‚19.39800294257108 disk.sda3.out‚454.8125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda.in‚0.203125( proc.memory.availableMB‚ 686.33984375#{žä-‚(:“ ëÜåÔÐÙµ gpu.0.gpu‚97.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚42.0 gpu.0.powerWatts‚124.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.895799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ÁUÈfå-‚(:Û ëÜåÔú–϶ disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚19.398785813610786 disk.sda2.out‚0 disk.rbd1.out‚ 16.0673828125 disk.sda.in‚0.203125 disk.sda.out‚ 455.47265625( proc.memory.availableMB‚ 623.58984375 network.recv‚ 4.113017e+06 disk.rbd1.in‚0 disk.drbd1716.in‚0$ memory_percent‚96.19390964508057! proc.memory.rssMB‚ 2955.671875* proc.memory.percent‚18.039989471435547$ disk./.usageGB‚289.0559272766113& disk.nvme0c0n1.out‚85449.150390625 disk.drbd2080.in‚0 network.sent‚ 3.10246e+07 disk.dm-1.out‚ 455.47265625 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚85449.150390625 disk.drbd2266.out‚0 disk.dm-0.in‚472.125 disk.drbd2265.out‚0 disk.sda2.in‚0! disk.dm-0.out‚85449.150390625 disk.sda3.out‚ 455.47265625 disk.sda1.out‚0 cpu‚25.273224943387397 proc.cpu.threads‚17 disk.nvme0n1.in‚472.125 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚472.125 disk.sda3.in‚0.203125éŒÓh]æ-‚(: úÜåÔµ´•µ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚3® Ã>3.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.438' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.88759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š¯ gç-‚(:Ü úÜåԲ᯶ disk.nvme0c0n1.in‚472.125& disk.nvme0c0n1.out‚85491.919921875* disk./.usagePercent‚19.399628846421038 disk.sda.out‚ 456.14453125 disk.nvme0n1.in‚472.125 disk.dm-1.in‚0.203125" proc.memory.rssMB‚ 2883.5390625 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0( proc.memory.percent‚17.5997257232666% disk./.usageGB‚289.06848907470703 disk.sda3.in‚0.203125 disk.sda3.out‚ 456.14453125 disk.drbd2080.in‚0 disk.sda.in‚0.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 16.1298828125 disk.dm-0.in‚472.125 network.recv‚ 4.116934e+06$ disk.nvme0n1.out‚85491.919921875 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0' proc.memory.availableMB‚ 695.8359375 disk.drbd2080.out‚0 disk.dm-1.out‚ 456.14453125 disk.drbd2266.in‚0 disk.sda1.out‚0$ memory_percent‚95.75295448303223 cpu‚25.278221762226007 network.sent‚ 3.1030448e+07! disk.dm-0.out‚85491.919921875r€»‘è-‚(:† ‰ÝåÔñÊ×´ gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚46.0 gpu.0.powerWatts‚283.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CëT‡jé-‚(:ß ‰ÝåÔÀÁ¬· disk.dm-1.out‚ 456.91796875& disk.nvme0c0n1.out‚85537.376953125 disk.sda.out‚ 456.91796875 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2955.80859375 disk.sda3.in‚0.203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2080.out‚0$ disk.nvme0n1.out‚85537.376953125( proc.memory.availableMB‚ 623.41015625 cpu‚25.282975712057585 network.sent‚ 3.1036845e+07 network.recv‚ 4.121376e+06 disk.dm-1.in‚0.203125! disk.dm-0.out‚85537.376953125 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚472.125 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 16.1298828125 disk.dm-0.in‚472.125 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚472.125 proc.cpu.threads‚17$ disk./.usageGB‚289.0716133117676 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.203125$ memory_percent‚96.19500637054443* disk./.usagePercent‚19.399838516591583 disk.sda3.out‚ 456.91796875 disk.drbd2080.in‚0* proc.memory.percent‚18.040823936462402šž.34ê-/ _wandbruntime‚12190  _runtime‚12190ç;;ƒ*ë-‡U cNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚1645125539583488 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3878632448 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1645125515289088 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚25849301173248 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 217119924 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 28510781440 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3938451456 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚991 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚12190.574684876 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3938451456 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2329606656 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1441415603791872 fQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚1616908620990464 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 21432077312 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 477681835 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18278 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 224997437 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 224997304 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 129126912 _JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚1642716518911051 % _timestamp‚1.7884402122192311e+09 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 260561911 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚25851630779904 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚133 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1670977170363392 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 477681835 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690840 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3296843776 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1670974816462336 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1670974816462336 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚25851630779904 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1670977170363392 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 129926024 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚131 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 111478272 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 477700118 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚25849301173248 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1405588823784448 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 260580189 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1405588841433088 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚35826650880512 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1616908597659648 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 24645479249 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 24572329984 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1878 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1670977170363392 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 477681835 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2327690840 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3296843776 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1670974816462336 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1670974816462336 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚25851630779904 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18278 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3180 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1302 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3180 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 21880356864 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27964 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 24572329984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18283 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1670977170363392 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 25086444032 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3180 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 129926024 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚131 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 111478272 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2351021656 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1878 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 477700118 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚25849301173248 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1405588823784448 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 28510781440 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27964 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 260580189 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 21880356864 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1405588841433088 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 425119232 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3876693096 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 28510781440 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3878632448 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚35826650880512 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1616908597659648 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚991 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28217ߨýØÀ4í-/ _wandbruntime‚12191  _runtime‚12191úKÚäšî-‘S ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1642801964962955 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878632448 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1671064907089408 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 225011602 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3296843776 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 260569605 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1645215132193280 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1645215107898880 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚133 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1405664100509696 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 260587883 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1645215107898880 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚25852128797184 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚25849799190528 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 260587883 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚14056641181583365”œ)ï-–S @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1878 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3878632448 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 25086444032 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚95076047 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 477721010 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 24572329984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3180 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 25086444032 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚95075916 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚35827260376064 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚35827148897792 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2351021656 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1645215132193280 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚25849799190528 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 217133127 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1302 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 129935688 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 260569605 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 217133122 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3329 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚738 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚971 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 260569605 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2353901056 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1645215132193280 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1645215107898880 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚133 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1405664100509696 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2327690840 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3876693096 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 260587883 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3329 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1645215107898880 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚25852128797184 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚25849799190528 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 260587883 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1405664118158336 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1642801964962955 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18278 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2329606656 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 21880356864 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1700 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 24645479249 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3180 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1878 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3878632448 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 28510781440 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1878 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27964 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1671064907089408 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27964 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 225011602 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3296843776 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 21880356864 B&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.27635136246681213 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22326262295246124 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5352296233177185 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_2_of_5‚0.45468512177467346 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9933088421821594 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006319267675280571 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3309137225151062 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41081106662750244 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35239464044570923 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.42645370960235596 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06564180552959442 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9478970170021057 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41013893485069275 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941194653511047 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014051341451704502 particle/accuracy_1_of_6‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3305555284023285 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4254273772239685 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32083335518836975 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_5‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6020553708076477 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.689393937587738 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5071877241134644 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01068117842078209 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23017241060733795 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1862470954656601 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015266861766576767 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990437388420105 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8193318843841553 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3581078350543976 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4466666877269745 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.257791668176651 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.40968072414398193 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.444444477558136 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5234895348548889 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.35547131299972534 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0026455027982592583 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918959140777588 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.160720556974411 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4320175349712372 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22076143324375153 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4344025254249573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23678743839263916 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5370047092437744 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.37926554679870605 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_7_of_7‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47331178188323975 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3650625944137573 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1581725925207138 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1624094396829605 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18978828191757202 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22099293768405914 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03242630511522293 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.062337636947631836 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5791667103767395 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931147694587708 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09382135421037674 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9875672459602356 particle/accuracy_4_of_4‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 trainer/global_step‚12089 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3422558605670929 ,jet/accuracy_4_of_4‚0.005646205972880125 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27129632234573364 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1666666716337204 particle/accuracy_6_of_6‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2945595383644104 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.455739289522171 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2512918710708618 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3006018400192261 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10342179983854294 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 6validation_average_jet_accuracy‚0.4005478322505951 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3436364233493805 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3468409478664398 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 jet/accuracy_5_of_7‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4108046293258667 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.550000011920929 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5137931108474731 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10504580289125443 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9932836294174194 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018298059701919556 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24927598237991333 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013725490309298038 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.35583022236824036 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34876543283462524 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4004417359828949 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.019265232607722282 particle/accuracy_3_of_5‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚I'|d»M0.5625 _step‚993 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1449333280324936 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39548608660697937 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23497022688388824 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5496031641960144 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.22987249493598938 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4041438400745392 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5074616074562073 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 _runtime‚12206.08274995 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3106791377067566 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04208263382315636 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.37058255076408386 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21547241508960724 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15139879286289215 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_6_of_6‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1894359141588211 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47002336382865906 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08337099105119705 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23275882005691528 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19684039056301117 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5423610806465149 'particle/accuracy‚0.846261203289032 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 'particle/f_score‚0.3761579394340515 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.32340654730796814 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43701550364494324 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3581078350543976 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 *jet/accuracy_2_of_6‚0.5833333134651184 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4022108316421509 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20750218629837036 ,jet/accuracy_3_of_3‚0.021152393892407417 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.28550538420677185 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27279359102249146 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3245614171028137 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1904343217611313 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26451101899147034 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.21360154449939728 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.553967297077179 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.48106062412261963 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3831791281700134 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35964709520339966 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2433394342660904 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1862470954656601 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3538959324359894 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3146551251411438 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27158477902412415 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0334293432533741 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2522960603237152 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37432435154914856 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.2150205820798874 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2952535152435303 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01880224235355854 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03682493418455124 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4717774987220764 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3309941291809082 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9933648109436035 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3963579833507538 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18833142518997192 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5301965475082397 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5667535066604614 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.41145971417427063 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3642856776714325 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.41783517599105835 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21716026961803436 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4622292220592499 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8419435620307922 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5597529411315918 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17847378551959991 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.991405189037323 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3442193269729614 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 +jet/accuracy_4_of_6‚0.01923076994717121 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7117940783500671 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 particle/accuracy_2_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4715900421142578 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10103807598352432 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_3‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9881848096847534 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6357486844062805 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2580414414405823 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_4‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 (loss/val_total_loss‚41.2071496171683 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04556650295853615 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3999577462673187 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.40648216009140015 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26146531105041504 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5286738872528076 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5678558349609375 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22305315732955933 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2078666239976883 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39080458879470825 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 particle/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4385802745819092 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.018435288220643997 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2202380895614624 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.29702165722846985 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3414076566696167 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25386831164360046 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20352217555046082 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3677268624305725 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_7‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39984193444252014 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2652498185634613 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3627352714538574 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5123456716537476 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20293210446834564 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4027777910232544 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.788440227725612e+09 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 epoch‚9 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_2_of_3‚0.21781814098358154 particle/accuracy_5_of_7‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 *jet/accuracy_1_of_4‚0.9008512496948242 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2813822627067566 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3926461637020111 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6982898116111755 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27938222885131836 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4097301661968231 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2883838415145874 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9425883293151855 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333332419395447 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11609703302383423 particle/accuracy_1_of_1‚1øù72ô-Ó¥ & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 trainer/global_step‚12089 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3422558605670929 ,jet/accuracy_4_of_4‚0.005646205972880125 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27129632234573364 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1666666716337204 particle/accuracy_6_of_6‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2945595383644104 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.455739289522171 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2512918710708618 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3006018400192261 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10342179983854294 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 6validation_average_jet_accuracy‚0.4005478322505951 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3436364233493805 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3468409478664398 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 jet/accuracy_5_of_7‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4108046293258667 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.550000011920929 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5137931108474731 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10504580289125443 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9932836294174194 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018298059701919556 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24927598237991333 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013725490309298038 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.35583022236824036 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34876543283462524 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4004417359828949 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.019265232607722282 particle/accuracy_3_of_5‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 _step‚993 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1449333280324936 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39548608660697937 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23497022688388824 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5496031641960144 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.22987249493598938 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4041438400745392 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5074616074562073 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 _runtime‚12206.08274995 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3106791377067566 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04208263382315636 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.37058255076408386 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21547241508960724 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15139879286289215 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_6_of_6‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1894359141588211 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47002336382865906 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08337099105119705 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23275882005691528 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19684039056301117 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5423610806465149 'particle/accuracy‚0.846261203289032 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 'particle/f_score‚0.3761579394340515 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.32340654730796814 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43701550364494324 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3581078350543976 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 *jet/accuracy_2_of_6‚0.5833333134651184 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4022108316421509 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20750218629837036 ,jet/accuracy_3_of_3‚0.021152393892407417 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.28550538420677185 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27279359102249146 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3245614171028137 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1904343217611313 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26451101899147034 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.21360154449939728 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.553967297077179 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.48106062412261963 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3831791281700134 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35964709520339966 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2433394342660904 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1862470954656601 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3538959324359894 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3146551251411438 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27158477902412415 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0334293432533741 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2522960603237152 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37432435154914856 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.2150205820798874 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2952535152435303 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01880224235355854 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03682493418455124 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4717774987220764 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3309941291809082 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9933648109436035 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3963579833507538 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18833142518997192 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5301965475082397 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5667535066tR·£`604614 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.41145971417427063 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3642856776714325 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.41783517599105835 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21716026961803436 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4622292220592499 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8419435620307922 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5597529411315918 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17847378551959991 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.991405189037323 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3442193269729614 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 +jet/accuracy_4_of_6‚0.01923076994717121 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7117940783500671 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 particle/accuracy_2_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4715900421142578 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10103807598352432 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_3‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9881848096847534 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6357486844062805 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2580414414405823 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_4‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 (loss/val_total_loss‚41.2071496171683 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04556650295853615 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3999577462673187 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.40648216009140015 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26146531105041504 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5286738872528076 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5678558349609375 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22305315732955933 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2078666239976883 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39080458879470825 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444465637207 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 particle/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4385802745819092 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.018435288220643997 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2202380895614624 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.29702165722846985 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3414076566696167 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25386831164360046 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20352217555046082 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3677268624305725 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_7‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39984193444252014 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2652498185634613 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3627352714538574 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5123456716537476 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20293210446834564 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4027777910232544 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.788440227725612e+09 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 epoch‚9 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_2_of_3‚0.21781814098358154 particle/accuracy_5_of_7‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 *jet/accuracy_1_of_4‚0.9008512496948242 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2813822627067566 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3926461637020111 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6982898116111755 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27938222885131836 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4097301661968231 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2883838415145874 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9425883293151855 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333332419395447 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11609703302383423 particle/accuracy_1_of_1‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3293028473854065 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3396890163421631 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08988095074892044 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4252205193042755 +jet/accuracy_2_of_4‚0.35016751289367676 particle/accuracy_2_of_5‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995679259300232 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40295425057411194 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9922780394554138 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02769336849451065 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35885417461395264 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.035133298486471176 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.27635136246681213 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22326262295246124 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5352296233177185 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_2_of_5‚0.45468512177467346 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9933088421821594 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006319267675280571 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3309137225151062 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41081106662750244 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35239464044570923 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.42645370960235596 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06564180552959442 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9478970170021057 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41013893485069275 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941194653511047 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014051341451704502 particle/accuracy_1_of_6‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3305555284023285 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4254273772239685 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32083335518836975 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_5‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6020553708076477 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.689393937587738 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5071877241134644 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01068117842078209 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23017241060733795 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1862470954656601 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015266861766576767 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990437388420105 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8193318843841553 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3581078350543976 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4466666877269745 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.257791668176651 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.40968072414398193 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.444444477558136 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5234895348548889 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.35547131299972534 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0026455027982592583 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918959140777588 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.160720556974411 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4320175349712372 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22076143324375153 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4344025254249573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23678743839263916 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5370047092437744 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.37926554679870605 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_7_of_7‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47331178188323975 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3650625944137573 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1581725925207138 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1624094396829605 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18978828191757202 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22099293768405914 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03242630511522293 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.062337636947631836 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5791667103767395 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931147694587708 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09382135421037674 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9875672459602356 particle/accuracy_4_of_4‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05áöÌ‘õ-‚(:† §ÝåԮ÷µ gpu.0.gpu‚9.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚33.82820129394531* gpu.0.memoryAllocatedBytes‚ 29058203648 gpu.0.temp‚35.0 gpu.0.powerWatts‚101.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06QªtTö-‚(:É §ÝåÔø‡¸ disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0" disk.nvme0n1.out‚ 85635.5703125 disk.nvme0n1.in‚472.125 disk.dm-1.out‚458.0625 proc.cpu.threads‚17) disk./.usagePercent‚19.40111164205936 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 cpu‚25.291840805343792 disk.dm-1.in‚0.203125 disk.drbd2266.in‚0 disk.dm-0.in‚472.125 disk.dm-0.out‚ 85635.5703125 disk.sda3.out‚458.0625 disk.drbd2266.out‚0 disk.rbd1.out‚ 16.255859375$ disk.nvme0c0n1.out‚ 85635.5703125% proc.memory.availableMB‚ 4.4921875* proc.memory.percent‚31.626439094543457 disk.drbd2265.in‚0 network.sent‚ 3.1150844e+07 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 5181.67578125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚472.125 disk.sda3.in‚0.203125$ memory_percent‚99.97258186340332 network.recv‚ 4.130978e+06% disk./.usageGB‚289.09058380126953 disk.sda.in‚0.203125 disk.sda.out‚458.0625ÍÜéR÷- gzvulb17j@ ±ÝåÔ€’¯•0`Trainer.fit` stopped: `max_epochs=10` reached. €º3Êžø- gzvulb17j‹ ´ÝåÔ ÀïzEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:19:00 • 0:00:00 1.06it/s v_num: lb17DGY¼$ù- gzvulb17j ´ÝåÔ°Õ‚ ,Š"$4ú-/ _wandbruntime‚12224  _runtime‚12224ãN–û-‚(’À_